Fun with AWS CLI - Cost Explorer (ce) API

By Tony Lee

If you are an avid Amazon Web Services (AWS) consumer and have been thinking about ways to integrate these services into third party-tools (such as CyBot - https://github.com/cylance/CyBot), you have come to the right place. In this article we will cover how to setup and use the aws2 client (AWS CLI) to easily interact with AWS' APIs.  Per Amazon:  "AWS Command Line Interface (AWS CLI) is an open source tool that enables you to interact with AWS services using commands in your command-line shell."  Our example below will illustrate just one possibility by using the Cost Explorer (ce) API determine your current and past AWS bills via the command line or third-party tool.

In other words, we are ditching the Web interface shown below:

Figure 1:  AWS Billing Web Interface

In favor of querying the information via the command line:

Figure 2:  AWS Billing query via command line

Or better yet, a ubiquitous tool such as your favorite chat application using CyBot:

Figure 3:  Enabling CyBot to query the AWS API

We will cover the following tasks to enable this integration:

• Configuring AWS Permissions
• Policy creation
• Group creation
• User creation
• Installing the aws2 client
• Configuring the aws2 client
• Using the aws2 client

Cost Estimation Requirements

Before we dive into the configuration steps, there are two main requirements:

• Access to the following endpoint:  https://ce.us-east-1.amazonaws.com
• aws2 client
• A while back, we tried version 1 of the aws client, but it lacked the cost estimator (ce) command, thus we had to upgrade to version 2 (both referenced via the aws command)

Configuring AWS Permissions

Note:  This step generates your AWS Access Key ID and AWS Secret Access Key.  Copy these down in a safe place because you will need this info in the AWS client setup section

First create and retrieve your AWS credentials via IAM:
https://console.aws.amazon.com/iam/home

Policy Creation
Once, authenticated, create a policy to allow only cost estimation queries (following the principle of least privilege it may be possible to restrict this even more, but that is an exercise left up to the reader):

1. Click "Customer Managed Policies"
2. Create policy button
3. JSON tab, and copy and paste the following:

{
"Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ce:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Figure 4:  Policy Creation via JSON

Group Creation
Now that the policy exists, create a Cost Estimator Group and assign it the newly created policy by performing the following steps:

1. Click "Groups"
2. Click Create New Group Button
3. Name the group: CostEstimatorAPI
4. Attach a policy:  CostExplorerPolicy
5. Next
6. Apply

User Creation

Now that the group exists, create a user and assign it to the newly created group by performing the following steps:

1. Click "Users"
2. Add user button
3. Name the user
4. Access type:  Programmatic access
5. Select the CostEstimatorAPI Group we created
6. Click next button
7. Click Create user button

Figure 5:  User is created and tied to the group that has cost estimator permissions

Installing the aws2 Client

We installed the aws client on an Ubuntu Linux VM.  If you are using a different operating system, feel free to follow the instructions contained in the following documentation:  https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

unzip awscliv2.zip

sudo ./aws/install


Now check the version, you might see something like the
/usr/local/bin/aws --version

You might see something like the following:
aws-cli/2.0.33 Python/3.7.3 Linux/4.4.0-31-generic botocore/2.0.0dev37

Note:  This needs to be version 2.x or higher for the cost explorer functionality to exist.

Configuring the aws2 Client

Note:  Have your AWS Access Key ID and AWS Secret Access Key ready from the Setup AWS Permissions section

aws configure

AWS Access Key ID [None]: AK[redacted]YY
AWS Secret Access Key [None]: vn[redacted][redacted][redacted]Rm
Default region name [None]: 
Default output format [None]: 

Using the aws2 Client

When using the aws client, you can use the following syntax to get help in trying to figure out how to achieve your goals.

  aws help

  aws <command> help

  aws <command> <subcommand> help

Be sure to also consult the docs for syntax help and example usage.  In our case, we are looking at cost explorer functionality:

https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ce/index.html#cli-aws-ce

Specifically for our example, we will look to retrieve the cost and usage information:
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ce/get-cost-and-usage.html

After a bit of trial an error to get the exact format of required parameters, we have the following:

aws ce get-cost-and-usage --time-period Start=2020-07-01,End=2020-08-01 --granularity MONTHLY --metrics "BlendedCost" "UnblendedCost" "UsageQuantity"

Which yields the following output:

{
    "ResultsByTime": [
        {
            "TimePeriod": {
                "Start": "2020-07-01",
                "End": "2020-08-01"
            },
            "Total": {
                "BlendedCost": {
                    "Amount": "xx.6272352241",
                    "Unit": "USD"
                },
                "UnblendedCost": {
                    "Amount": "xx.6272352241",
                    "Unit": "USD"
                },
                "UsageQuantity": {
                    "Amount": "xxx.1172177568",
                    "Unit": "N/A"
                }
            },
            "Groups": [],
            "Estimated": true
        }
    ]
}

Two things to note about the time period parameter (--time-period Start=2020-07-01,End=2020-08-01)

1. It would be ideal if the aws client accepted terms such as:  "This year", "this month", "this week", or "today", but now that this is integrated into our third party tool, we can create that logic.
2. But at least the client allows you to select a date in the future (ex:  The first of next month) for calculating the present time period.

Conclusion

We hope this article serves as a quick introduction and jumpstart to your interactions with the AWS API using the freely available client. While not perfect, this client is quite powerful and can be used to quickly interact with the Amazon Web Services API to build integrations into third-party tools.  Feel free to leave any comments in the section below.  Happy Hacking.

References

These are references which we found to be useful:

• https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html
• https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ce/index.html
• https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ce/index.html#cli-aws-ce
• https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ce/get-cost-and-usage.html

Building a DirtyNet - The Ultimate Malware Playground

By Tony Lee

Do you have a safe place to handle and investigate potential malware? Maybe you are you doing this on your home PC or in a VM to try to protect the rest of your network?  Whether you have a DirtyNet or not, let’s start an open discussion that covers some safe malware handling possibilities along with the pros and cons of each. Maybe this will help provide justification or a framework for even a minimal amount of funding to build something official and supported.

Figure 1:  DirtyNet Information Flow

 

The Benefits

There are many benefits to building a DirtyNet to take a closer look at potential malware, here are a few:

• Determine maliciousness of malware
• Never seen before binaries – Threat intel and malware repositories know nothing about it
• Low conviction rate – Threat intel and repositories have seen it, but are not convinced of the maliciousness

• Extract Indicators of Compromise (IOCs)
• Useful in hunting for presence in the rest of the environment

• Protect the rest of the network
• Prevent users from mishandling malware on their corporate PCs by providing a dedicate environment

 

Possible Solutions

There are probably a large number of potential solutions – many of them bad ideas and some good ideas. We will cover the two most common setups here:

1) Stand-alone Malware Analysis Laptops

This is very common because it is easy to implement and somewhat reduces the risk of accidental malware spread.  There are some pros and cons that should be noted though:

Pros

• Lower risk of mishandled malware than using a VM on your corp machine

Cons

• Requires sneaker net – USB drives with write-blockers
• Not real-time feeding from sensors
• Difficult to connect/automate sensors and analysis stack
• Difficult to scale to a large amount of malware
• Difficult to extract and harvest IOC for reuse

 

2) Building a DirtyNet

This is not always the route chosen, but when built correctly, it is a beautiful and graceful solution. Here are the pros and cons:

Pros

• Can be automated
• Flexible sensor integration
• Flexible automated malware analysis stack
• Scales to higher volume of malware

Cons

• Higher risk of mishandled malware when not following proper procedures

 

DirtyNet Rules

Notice the single con of the DirtyNet…  But just like fight club, it is important that EVERYONE follow the rules:

• Write-only file share exposed to larger environment to accept malware and move inside DirtyNet
• Malware stays within DirtyNet.  No outbound traffic except whiteline for FireEye AX / Malware analysis stack
• Malware can be taken from the Malware repo and moved to an analysis workstation within DirtyNet
• Malware is ideally password protected zipped with “infected” before transfer and during storage
• Analyst connection to dirty net does not compromise the rest of the network. Virtualized desktops through a browser plugin with no file sharing is a decent option.

 

Requirements

So you want to build a DirtyNet now... and you are curious as to what you might need. The info below contains the high-level requirements:

• Hardware or Virtual (AWS/Azure/GCP) capacity 
• Whiteline randomized POP for malware to make callouts (ideal but not necessary)
• Viper or similar framework for the Malware repository
• Time from a network engineer to set up network and hosts

Conclusion

Now that you have the blueprint, will you be building a DirtyNet of your own? Also, if you already have a DirtyNet, we would be interested in hearing your tips, tricks, or opinions posted in the comment section below. As a bonus, do you have any cool integrations setup?  For example, we have CyBot’s (https://github.com/cylance/CyBot) Cuckoo plugin safely feeding out instance with malware. Thanks for reading!

Figure 2:  CyBot Cuckoo plugin safe submission to DirtyNet