How To Fix Exposed AWS Credentials

Posted by Ben Potter on Thursday, November 19, 2020

Contents

aws

Have you lost your AWS access key or password?

In AWS, there are different types of identities that you can use to login to the console, and access API’s for each service. Identities include the root user, IAM users, IAM groups, and IAM roles. Each identity except for IAM groups have credentials that you must protect by following IAM best practices and Well-Architected security best practices. This article will help you understand how you can fix and recover from exposing your AWS credentials.

Root User Credentials

When you first create an AWS account you start with the root user which is the email address you signed up for that account with. The root user has full access to everything including resources and data in the account, and it can even close the account. The root user has a password to access the console, and optionally access keys (access key IDs and secret access keys) for accessing the API’s. You should never have access keys for your root user, you should delete or deactivate them immediately. If you only have a password for the console, then you should enable multi-factor authentication.

If you have lost access keys for your root user, and can still login to the console, you can delete or deactivate them. As soon as you have done that, reset your password, enable MFA if you haven’t already, then follow the investigation section to check what resources might have been created and other actions.

If you have lost or forgotten the password for your root user, you need access to the email address that you signed up with. Go to the AWS Management Console and follow the link to reset your password. You will receive an email from an address ending in @amazon.com or @aws.amazon.com with instructions. If you don’t have access to your email address, then you can follow the instructions to either re-create the email address or alias, or contact AWS support for other options. You should then follow the investigation section to check what resources might have been created and other actions.

IAM User Credentials

IAM users are used to access your AWS account using a password for the console, or access keys for the API’s. You can configure users for either people, who could use both the console and API’s, or things that need access keys like a component in your application. Note that you should never have access keys in your code, there are much more secure methods which are covered in Well-Architected security pillar. It is a best practice defined by Well-Architected to centralize your identities, e.g. by using AWS Single Sign-On so you do not need IAM users, and keys can be generated for you that are temporary.

If an IAM user console password is lost, you can simply reset it in the IAM console using a different user that has permission to. If that was your only IAM user you will need to login as your root user to reset it in the IAM console. Unlike the root user which has full access to your account, an IAM user must have a policy attached, or be a member of a group with a policy. The policy is what grants permission to the user, if the policy had administrator permissions it could have created more IAM users or keys, and many other actions. You should then follow the investigation section to check what resources might have been created and other actions.

If an IAM access key is lost, you can disable it, then delete it, in the IAM console with any user that has permissions:

  1. Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.
  2. In the navigation pane, choose Users.
  3. Choose the name of the user of the access keys you want to manage, and then choose the Security credentials tab.
  4. In the Access keys section, disable an active access key by choosing Make inactive.
  5. I recommend you only disable the access key until you have confirmed it’s not being used for anything critical, and you will see access denied attempts in your CloudTrail logs until its deleted. This provides you more information into who or what is trying to use it.
  6. To delete an access key, choose its X button at the end of the row. When prompted for confirmation, choose Delete.

You should then follow the investigation section to check what resources might have been created and other actions.

Investigate Misuse of Compromised AWS Credentials

It’s critical that you investigate how your AWS credentials were misused in your account. For example, if you accidentally committed access keys on GitHub, then many bots will have them in seconds. Your account could be used for Bitcoin mining by an attacker launching as many EC2 instances as they can across all regions to try and make money (costing you money). Your account could also be used for launching attacks across the internet. The amount of effort you put into the investigation depends on what permission the user had in your account. If you lost your root access keys, then the attacker could have done anything - create IAM users so they can get back in even if you disabled the keys, launch many instances in all regions, and more. The only thing that would help is if the account was a member of an AWS Organization and you have service control policies that block actions.

You should look at AWS CloudTrail Event History for suspicious events in the last 90 days in your CloudTrail Console. It is recommended that you create a trail in CloudTrail and configure the logs in CloudWatch Logs and/or Amazon S3 when you first setup your account. This allows you to query the logs in detail, and store them for longer than 90 days - only for events that occur from the point in time that you create the trail. I’ve created a basic hands-on lab for investigating CloudTrail events that you can find here. It includes AWS Command Line Interface (CLI) commands to query CloudWatch Logs Insights. You can search for events by IAM access key, IP address, and customize the queries yourself. I’ve also created a more advanced version which uses Jupyter notebooks written in Python which you can find here.

The actions that you see in the CloudTrail logs should dictate your response and further investigation. If an IAM user had full permission (read & write access) to Amazon S3, then they could have accessed or tampered with any objects (files) in S3 - including your CloudTrail logs. You should then investigate if a data breach has occurred by looking at S3 logs, or log an AWS support request for assistance. If your misused credentials had permission to IAM, e.g. the AdministratorAccess example manage policy, then the attacked could have created new IAM users, and access keys, to re-gain access to your account even though you have disabled the primary method of them gaining access. If you need further assistance I strongly encourage you to log an AWS support ticket, and if you do not already have business or higher support plan then select business support plan for 24x7 phone, email and chat access.