Fun with AWS CLI - Cost Explorer (ce) API

By Tony Lee

If you are an avid Amazon Web Services (AWS) consumer and have been thinking about ways to integrate these services into third party-tools (such as CyBot - https://github.com/cylance/CyBot), you have come to the right place. In this article we will cover how to setup and use the aws2 client (AWS CLI) to easily interact with AWS' APIs.  Per Amazon:  "AWS Command Line Interface (AWS CLI) is an open source tool that enables you to interact with AWS services using commands in your command-line shell."  Our example below will illustrate just one possibility by using the Cost Explorer (ce) API determine your current and past AWS bills via the command line or third-party tool.

In other words, we are ditching the Web interface shown below:

Figure 1:  AWS Billing Web Interface

In favor of querying the information via the command line:

Figure 2:  AWS Billing query via command line

Or better yet, a ubiquitous tool such as your favorite chat application using CyBot:

Figure 3:  Enabling CyBot to query the AWS API

We will cover the following tasks to enable this integration:

• Configuring AWS Permissions
• Policy creation
• Group creation
• User creation
• Installing the aws2 client
• Configuring the aws2 client
• Using the aws2 client

Cost Estimation Requirements

Before we dive into the configuration steps, there are two main requirements:

• Access to the following endpoint:  https://ce.us-east-1.amazonaws.com
• aws2 client
• A while back, we tried version 1 of the aws client, but it lacked the cost estimator (ce) command, thus we had to upgrade to version 2 (both referenced via the aws command)

Configuring AWS Permissions

Note:  This step generates your AWS Access Key ID and AWS Secret Access Key.  Copy these down in a safe place because you will need this info in the AWS client setup section

First create and retrieve your AWS credentials via IAM:
https://console.aws.amazon.com/iam/home

Policy Creation
Once, authenticated, create a policy to allow only cost estimation queries (following the principle of least privilege it may be possible to restrict this even more, but that is an exercise left up to the reader):

1. Click "Customer Managed Policies"
2. Create policy button
3. JSON tab, and copy and paste the following:

{
"Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ce:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Figure 4:  Policy Creation via JSON

Group Creation
Now that the policy exists, create a Cost Estimator Group and assign it the newly created policy by performing the following steps:

1. Click "Groups"
2. Click Create New Group Button
3. Name the group: CostEstimatorAPI
4. Attach a policy:  CostExplorerPolicy
5. Next
6. Apply

User Creation

Now that the group exists, create a user and assign it to the newly created group by performing the following steps:

1. Click "Users"
2. Add user button
3. Name the user
4. Access type:  Programmatic access
5. Select the CostEstimatorAPI Group we created
6. Click next button
7. Click Create user button

Figure 5:  User is created and tied to the group that has cost estimator permissions

Installing the aws2 Client

We installed the aws client on an Ubuntu Linux VM.  If you are using a different operating system, feel free to follow the instructions contained in the following documentation:  https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

unzip awscliv2.zip

sudo ./aws/install


Now check the version, you might see something like the
/usr/local/bin/aws --version

You might see something like the following:
aws-cli/2.0.33 Python/3.7.3 Linux/4.4.0-31-generic botocore/2.0.0dev37

Note:  This needs to be version 2.x or higher for the cost explorer functionality to exist.

Configuring the aws2 Client

Note:  Have your AWS Access Key ID and AWS Secret Access Key ready from the Setup AWS Permissions section

aws configure

AWS Access Key ID [None]: AK[redacted]YY
AWS Secret Access Key [None]: vn[redacted][redacted][redacted]Rm
Default region name [None]: 
Default output format [None]: 

Using the aws2 Client

When using the aws client, you can use the following syntax to get help in trying to figure out how to achieve your goals.

  aws help

  aws <command> help

  aws <command> <subcommand> help

Be sure to also consult the docs for syntax help and example usage.  In our case, we are looking at cost explorer functionality:

https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ce/index.html#cli-aws-ce

Specifically for our example, we will look to retrieve the cost and usage information:
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ce/get-cost-and-usage.html

After a bit of trial an error to get the exact format of required parameters, we have the following:

aws ce get-cost-and-usage --time-period Start=2020-07-01,End=2020-08-01 --granularity MONTHLY --metrics "BlendedCost" "UnblendedCost" "UsageQuantity"

Which yields the following output:

{
    "ResultsByTime": [
        {
            "TimePeriod": {
                "Start": "2020-07-01",
                "End": "2020-08-01"
            },
            "Total": {
                "BlendedCost": {
                    "Amount": "xx.6272352241",
                    "Unit": "USD"
                },
                "UnblendedCost": {
                    "Amount": "xx.6272352241",
                    "Unit": "USD"
                },
                "UsageQuantity": {
                    "Amount": "xxx.1172177568",
                    "Unit": "N/A"
                }
            },
            "Groups": [],
            "Estimated": true
        }
    ]
}

Two things to note about the time period parameter (--time-period Start=2020-07-01,End=2020-08-01)

1. It would be ideal if the aws client accepted terms such as:  "This year", "this month", "this week", or "today", but now that this is integrated into our third party tool, we can create that logic.
2. But at least the client allows you to select a date in the future (ex:  The first of next month) for calculating the present time period.

Conclusion

We hope this article serves as a quick introduction and jumpstart to your interactions with the AWS API using the freely available client. While not perfect, this client is quite powerful and can be used to quickly interact with the Amazon Web Services API to build integrations into third-party tools.  Feel free to leave any comments in the section below.  Happy Hacking.

References

These are references which we found to be useful:

• https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html
• https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ce/index.html
• https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ce/index.html#cli-aws-ce
• https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ce/get-cost-and-usage.html