Secure your local AWS Credentials with AWS Vault

In this post, I’ll go through the process of setting up IAM users according to AWS best practices and show how to secure the Access and Secret keys on our local machines to prevent them from various attack vectors. The main tool I will be using is called the AWS Vault. I’ll also show how to make using this tool easier with my setup of iTerm2 and oh-my-zsh.

When you configure your IAM user using the AWS CLI, it stores your credentials in plain text on your machine by default in an \.aws directory on your home directory. This is common knowledge and can be exploited in cases where your computer gets compromised and all of this can happen without your knowledge. With your credentials, an attacker can use them to do almost anything in your AWS account based on the permissions set on the associated IAM user.

AWS best practices states that you should never use your main account (the account that you used to sign up for AWS) to do any work. Instead, you should create an IAM admin user and give admin access needed for administrative tasks. Also other IAM users and roles should be created with limited access scoped to the tasks performed by these users and roles. This recommendation is widely practiced throughout the industry, but most engineers leave their credentials stored in plain text on their computers. Remember that these IAM admin accounts have full access to your AWS account and in the wrong hands can cause a lot of damage.

Attack vectors can include a stolen laptop, malware that you are not aware of in your computer, and a compromised npm package just to name a few. Each of these threats increases with each new member you add to your team.

AWS Vault securely stores and access AWS credentials in our local setup using our operating system’s secure keystore. It then generates temporary credentials from the keystore and exposes it to our shell and applications, making sure we do not store these credentials in plain text.

AWS Vault Installation

Use this link to find the installation instructions of AWS Vault for your operating system. I use OS X as my primary operating system and the command for my system is:

brew install --cask aws-vault.

Setup a New Profile

After creating a new IAM user in the AWS Management console, you are presented with a screen with both the Access key ID and Secret access key. These credentials are shown only once and you cannot retrieve them from the AWS console after this step. It is at this point where we will use the AWS Vault tool to securely store these access credentials on our own machine.

Run the following command to create a new profile for a user we will call s3-read-user. This is an example where we are simulating the process of creating an IAM user that has READ ONLY access to S3. The policies you associate with the IAM user should reflect the level of access you want to give to your users. Always use the principle of least privilege access:

# Use the values from the AWS console for the prompts below:
$ aws-vault add s3-read-user
Enter Access Key ID: aaaaaaa-aaaaaa
Enter Secret Key: bbbbbbb-bbbbbbb

If this is your first time using the AWS Vault tool, it will prompt you to setup a new password for the aws-vault keystore.

The above command sets up a new profile. Visiting the ~/.aws directory and opening the config file, you see the profile name listed. But when you open up the credentials file, it is empty. Previously, it listed the Access and Secret key stored in plain text. The AWS Vault tool stored our credentials encrypted in the OS’s keystore instead of storing it in plain text inside the credentials file.

Using the AWS Vault: Basic Commands

Now that we have our profile setup using AWS Vault, there are several new commands we need to know and use when we want to interact with our AWS account from the CLI.

The first one is the exec command which will create temporary credentials that are passed to other commands such as the AWS CLI command. In our case, suppose we have a couple of S3 buckets in our account and want to list all of them, execute the following command passing in the profile name s3-read-user:

$ aws-vault exec s3-read-user -- aws s3 ls
account_bucket-a
account_bucket-b
account_bucket-c

We can also use the AWS Vault tool to login to an IAM account that is setup with AWS Console login access using the following command:

$ aws-vault login <iam-login-account>

To show a list of all profiles in your computer, including ones that were not stored with the AWS Vault tool, run the following command:

$ aws-vault list

There are other commands to remove profiles, rotate credentials, clear temporary credentials from the secure keystore and more. I believe the above ones should be enough to get you started. Also this tool works with Roles and MFA (multi-factor authentication).

Integration with ZSH and oh-my-zsh

I use the iTerm2 terminal configured with the ZSH shell. I also use the oh-my-zsh framework for my ZSH shell. This should work in any terminal that uses ZSH not just iTerm2. So any Unix-like operating system, MacOS, Linux, BSD and WSL2 on Windows. ‘Oh My Zsh’ is an open source community-driven framework for managing your ZSH configurations. Think of it as a community supported alias package for your terminal.

‘Oh My Zsh’ supports plugins and the plugin we will add is called zsh-aws-vault. It provides many convenient aliases to use when running commands that interact with the configured AWS profiles.

The first step in installing the above plugin is to clone and download it to the plugins directory:

$ cd ~/.oh-my-zsh/custom/plugins
$ git clone [email protected]:blimmer/zsh-aws-vault.git

Next, edit your ~/.zshrc file and add the new plugin to the plugins line. We will be adding a plugin for AWS Vault. So edit your plugins line to look something like this: (Here, I’m adding AWS Vault in addition to git and dotenv plugins):

plugins=(
  git
  dotenv
  zsh-aws-vault
)

Usage with zsh-aws-vault

Here are the aliases that are added after you install the plugin for zsh:

We will use the following command to run the example above where we listed all S3 buckets in our account:

$ ave s3-read-user -- aws s3 ls
account_bucket-a
account_bucket-b
account_bucket-c

Sometimes you might not want to type extra commands each time you want to interact with your AWS account through the CLI. In such cases use the nifty avsh command to create a shell for a given profile. Then any command you run in that shell will use the associated profile. Using the example above, we can create a shell with the s3-read-user profile, then run any command without using the ave or aws-vault prefixes like we did above:

$ avsh s3-read-user

Now any command you run in that shell will use the associated profile from the previous command:

$ aws s3 ls
account_bucket-a
account_bucket-b
account_bucket-c

If you run the env command at this point, you will see all the AWS environment variables created in your terminal. You can use the exit command to go back to your regular shell and remove these variables from your shell environment.

$ env | grep AWS
AWS_VAULT=s3-read-user
AWS_DEFAULT_REGION=us-east-1
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=%%%
AWS_SECRET_ACCESS_KEY=%%%
AWS_SESSION_TOKEN=%%%
AWS_SECURITY_TOKEN=%%%
AWS_SESSION_EXPIRATION=2020-12-29T11:00:00Z

To exit from the temporary shell and back to your normal shell without the AWS environment variables, use the exit command:

$ exit

Add current AWS Profile to Terminal Prompt

The above recommendation and tools to secure AWS credentials works great and with practice can easily become second nature to anyone who uses it consistently. The only issue I have with this setup is that when we have populated our shell with AWS environment variables, we cannot easily tell which profile is currently being used. To fix this issue, the zsh-aws-vault plugin for ‘Oh My Zsh’ has a prompt function called prompt_aws_vault_segment. Adding this function to your shell prompt will show the current profile whose credentials are in your terminal environment variables.

In my setup, I use a theme that comes with ‘Oh My Zsh’ called miloshadzic.zsh-theme located in ~/.oh-my-zsh/themes. To modify the prompt exposed by the theme, I copied it to the custom themes folder in ~/.oh-my-zsh/custom/themes, and modified the theme file adding a right prompt by adding the following to the end of that file:

RPROMPT='$(prompt_aws_vault_segment)'

Now, when I use the avsh command, I can tell which profile is being used in the shell by looking at the right prompt. This is very useful and helps me avoid using the wrong profile when running commands in my AWS account.

Rotate Access and Secret Keys

Since the access keys are long-term keys, it is recommended to rotate them periodically. The AWS Console will start showing you warnings after a while if your keys have not been rotated. AWS Vault takes the pain out of rotating keys with this simple command:

$ aws-vault rotate s3-read-user -n

# If using the plugin, use the av alias
$ av rotate s3-read-user -n

Usage with AWS Toolkit for Visual Studio Code

The AWS Toolkit for VSCode is an extension used for working with AWS Services in VSCode. It can make calls to your AWS account and run AWS CLI commands for you. However, it discovers your profiles and uses secrets from the ~/.aws/credential file. Now if you remember, we don’t have anything in that file because AWS Vault has all our secrets encrypted in our keystore. Hence this tool cannot use our profiles to make authenticated calls to AWS.

However, the AWS CLI config supports sourcing credentials directly from an external process, using credential_process. We can use this to expose our master credentials (instead of the AWS Vault STS credentials). These credentials can then be used by the AWS Toolkit extension. Add the credential_process underneath the profile in the ~/.aws/config file.

[profile s3-read-user]
credential_process = aws-vault exec s3-read-user --json

Now, when you open VSCode with the Toolkit extension installed, you can run the AWS: Connect to AWS command and select the s3-read-user profile. The AWS Vault utility will ask for your keystore password and open the profile for use in VSCode.

Note that if you’re using credential_process in your config you should not use aws-vault exec on the command line to execute commands directly. The AWS SDK executes aws-vault for you when you supply a profile with your AWS CLI commands. You can still create a temporary shell with your profile using the avsh <profile> command and use it to run your AWS CLI commands.

Caveats To Know

AWS Vault issues temporary credentials and these are restricted from using some STS and IAM APIs. To overcome these restrictions, AWS requires these session tokens to be MFA authenticated. To enable MFA for a profile, specify the mfa_serial in ~/.aws/config.

If you don’t want to setup MFA, another way to bypass these restrictions is to use the --no-session flag when running commands that interact with IAM. Use the above flag if you get an error that states the security token is invalid (The security token included in the request is invalid (Service: AmazonIdentityManagement; Status Code: 403; Error Code: InvalidClientTokenId; Request ID: 37b49179-ca46-433a-bdaa-044cd5a36c88; Proxy: null)). Use the following command to pass the --no-session tag before executing your commands:

# Deploying infra using the AWS CDK.
$ aws-vault exec s3-read-user --no-session -- npx cdk deploy

# Using the zsh-aws-vault plugin
$ ave s3-read-user --no-session -- npx cdk deploy