Olivia A. Gallucci

Found a leaked API key? Deleting it from your code isn’t enough. The key still works, and it lives on in Git history, forks, and clones. You gotta fix key rotation. This post explains why, and what else to do after a leak.

Overview 

The first half of this post is a primer on APIs, authentication, authorization, and how keys leak. If you already know that ground, skip straight to key rotation.

APIs

APIs are sets of rules and protocols that let software applications communicate with each other, share data and functionality, and integrate with other systems and services.

For example, somebody could use APIs to learn about a GitHub profile.

# Use 'pip install requests' before running. 
import requests

# replace 'YOUR_GITHUB_USERNAME' with your GitHub username
github_username = 'YOUR_GITHUB_USERNAME'

# github API endpoint to retrieve user information
api_url = f'https://api.github.com/users/{github_username}'

try:
    # send a GET request to the GitHub API
    response = requests.get(api_url)

    # check if the request was successful (status code 200)
    if response.status_code == 200:
        # parse the JSON response
        user_data = response.json()

        # print some user information
        print(f"github username: {user_data['login']}")
        print(f"name: {user_data['name']}")
        print(f"bio: {user_data['bio']}")
        print(f"followers: {user_data['followers']}")
        print(f"following: {user_data['following']}")
        print(f"public repositories: {user_data['public_repos']}")
    else:
        # print an error message if the request was not successful
        print(f"request failed with status code {response.status_code}")
except requests.exceptions.RequestException as e:
    # handle any exceptions that may occur during the request
    print(f"an error occurred: {e}")

Gist

Other examples 

There are tons of other uses for APIs. 

Problems with APIs

The public uses many APIs, but some APIs are for a specific group of people. For example, a banking app’s API lets customers check their account balances and make transactions securely.

Unfortunately, malicious actors sometimes obtain unauthenticated or unauthorized access to APIs, leading to data breaches, data manipulation, denial-of-service attacks, and SQL or code injection attacks.

Authentication

Authentication is an act, process, or method of showing something (such as an identity, a piece of art, or a financial transaction) to be real, true, or genuine : the act or process of authenticating something.

Merriam-Webster 

According to Fortinet, “the user must present physical or nonphysical evidence (information) to the authentication platform to confirm the user’s identity.” Fortinet divides these confirmation practices into three categories: 

• What they have: The possession of a physical object, such as a key, keycard, key fob, or swipe card.
• What they know: Information that only the user would know, including a password, passcode, personal identification number (PIN), date of birth, Social Security number, or other personally identifiable information (PII).
• Who they are: Biometrics, or the use of an index finger, thumb, hand, voice, retina, face, or another unique physical identifier to gain access to a resource. The physical attribute must match what was used at the time of the user’s enrollment in the system.

For this blog post, the main method of authentication referenced is “what [the user] knows” (e.g., password-based authentication). With this form of authentication, an attacker could exploit an organization’s weak password policies by using brute-force or credential stuffing to access a user’s account.

Read more: CIA Triad: Confidentiality, Integrity, Availability. 

Authorization

Authorization “expresses permissions and rights granted to a user” (TrendMicro). It answers the question, “what is a [user/system/group] allowed to do?”

For instance, a manager granting an employee permission to access confidential company financial data, while denying access to other employees.

When authorization issues are present, attackers can perform actions they should not be able to. For instance, once inside a user’s account through legitimate authentication, an attacker might manipulate the API request to access another user’s account or attempt to execute transactions without proper authorization, exploiting vulnerabilities in the authorization process.

Keys

To combat unauthenticated and unauthorized access to data and services, APIs often require keys—sometimes called “API keys” or “authentication tokens.”

Each key is a piece of information, typically a string of characters or bits, used to encrypt, decrypt, or authenticate data in cryptographic systems.

Authentication code

Here is an example HTTP request with an “Authorization” header for basic authentication. The “Authorization” header contains the word “Basic” followed by a base64-encoded username and password. This is a basic authentication scheme where the server can decode the credentials to authenticate the client.

GET /api/resource HTTP/1.1
Host: example.com
Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==

Gist

Below is an example of a full HTTP request with an Authorization header containing an API key or access token.

GET /api/resource HTTP/1.1
Host: example.com
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

Gist

• GET /api/resource is the requested HTTP method and the endpoint.
• Host: example.com specifies the hostname of the server.
• Authorization: Bearer … is the Authorization header containing the API key or access token

Here, keys make sure only authorized users or applications can access services and data. Keys also allow for tracking usage for security and billing purposes.

The table below details how APIs commonly use keys. 

Common API key usage 

APIs use keys or tokens to verify the identity of users or applications and control their access to specific resources. The choice of key or token mechanism depends on the API’s security requirements and how complex the authorization process is.

Attack and remediation examples 

Without keys or proper authentication, APIs are open to unauthorized access, abuse, data breaches, and overuse of resources.

Several types of attacks can occur when APIs do not use proper authentication mechanisms such as API keys, including:

Authentication mechanisms like API keys, OAuth tokens, or other access controls reduce these risks.

Leaky keys

API keys handle authentication and authorization for many online services and resources, so exposed API keys are a security risk. Below are common mistakes that lead to API key leaks. 

• Storing keys in code repositories
  • Developers often mistakenly commit API keys directly into source code repositories, where they can be easily accessed by anyone with access to the repository’s history.
• Poor access control
  • Failing to restrict access to API keys or granting excessive permissions can result in unintended users gaining access to these keys.
• Lack of encryption
  • Transmitting or storing API keys without encryption makes them susceptible to interception by attackers during data transmission or storage.
• Hardcoding keys
  • Embedding API keys directly into applications or scripts without utilizing secure storage methods can lead to inadvertent exposure.
• Inadequate monitoring and auditing
  • Failing to monitor API key usage and unusual activity can delay the detection of breaches or unauthorized access.

Keys scanning

Developers leak API keys often, so several products and services now help with this. One is TruffleHog, an open-source tool that searches the commit history of Git repositories for high-entropy strings, which often point to sensitive information like API keys, passwords, and other secrets.

TruffleHog

TruffleHog scans the commit history of a Git repository and analyzes each commit for high-entropy strings, which might be sensitive data.

A string with high entropy contains a seemingly random arrangement of characters, making it difficult to predict or compress. High-entropy strings can be detected by their lack of discernible patterns or repetitions, like  “R#t$5&@pL!9KzWq2*X.” 

Here is an example of a high entropy string found by TruffleHog (specifically, the raw result value “AKIAYVP4CIPPERUVIFXG”): 

$ trufflehog git https://github.com/trufflesecurity/test_keys --only-verified
🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

Found verified result 🐷🔑
Detector Type: AWS
Decoder Type: PLAIN
Raw result: AKIAYVP4CIPPERUVIFXG
Line: 4
Commit: fbc14303ffbf8fb1c2c1914e8dda7d0121633aca
File: keys
Email: counter <counter@counters-MacBook-Air.local>
Repository: https://github.com/trufflesecurity/test_keys
Timestamp: 2022-06-16 10:17:40 -0700 PDT
…

Commit Source

By running TruffleHog on a codebase, developers and security engineers can find and remove exposed secrets from repositories before attackers use them.

Why API keys and TruffleHog? 

My friend contacted me and asked if I would be interested in contributing to How To Rotate  (HTR), an open-source collection of API key rotation tutorials by Truffle Security. 

My friend sent me two links on how I could start: 

• How to Rotate Leaked API Keys by Truffle Security 
• How to Rotate: Key Rotation Tutorials by Joe Leon

I do not know much about APIs or key rotations, so contributing to HTR is a good way to learn. It also helps other beginners: my step-by-step posts come from a beginner mindset and include details an expert might overlook.

HTR maintainers review all of my contributions for accuracy.

Outside of TruffleHog, other tools help detect high-entropy strings, like Gitrob (archived), Gitleaks (creator works for Truffle Security), and GitGuardian (commercial tool). I am focusing on TruffleHog because its community is supportive and responsive, which matters when you rely on others for maintenance and long-term support. Truffle Security is also dedicated to open-source software, collaboration, and transparency. 

In addition, it makes me happy to contribute to a project my friends made.

Scanning limitations and remediation attempts 

Scanners like TruffleHog find keys, but they don’t stop anyone from using them. A key a scanner finds can still work. 

Most teams revoke or invalidate a leaked key as soon as they find it. Both are often tedious, slow, and error-prone.

Revoke vs invalidate 

“Revoke” and “invalidate” both make a key or credential unusable, but they differ slightly. 

Revocation

Revocation is a deliberate act: an authority, such as a system administrator or security team, cancels a key and often tells systems and users it is no longer valid.

In practice, that means disabling the key in the systems that use it and issuing a replacement.

Invalidation

Invalidation is the broader term: any way a key becomes unusable, including revocation. Key expiration, password changes, or security breaches can all invalidate a key.

Either way, the systems that used the key no longer accept it.

In practice, people use the two terms interchangeably. Both mean the key is no longer trusted for authentication or encryption.

Pitfalls of revoking without key rotation

Revoking or invalidating an API key can break legitimate services that still use it and cause downtime, which hits users and revenue. 

That’s why teams use key rotation instead of, or alongside, revocation and invalidation. Rotation replaces the old key with a new one, so existing systems keep working with minimal disruption and no rushed manual reconfiguration.

Key rotation

Key rotation means replacing cryptographic keys to reduce the risk from compromised or outdated keys.

In Leon’s blog post, he mentions that: 

Security engineers and developers run our open-source secret scanner, TruffleHog, to find and remediate leaked secrets across their Software Development Life Cycle. While TruffleHog can detect over 750 types of leaked secrets, that only solves part of the problem. Users often seek our advice on remediation.

With very, very few exceptions, the most effective way to remediate a leaked secret is with key rotation. Simply deleting the code that exposes the key or editing the Git history is insufficient.

But we are left wondering why “deleting the code that exposes the key or editing Git history is insufficient.” 

The big question is: 

Outside of the pitfalls of invalidating and revoking keys, why is key rotation needed? 

Why key rotation? 

Deleting the code that exposes the key or editing Git history helps, but neither is enough on its own. 

1. Incomplete removal

When a secret is exposed in code, it may not be limited to a single file or commit. Secrets can be spread across multiple code files, configuration files, or external services. Deleting one instance of the exposed key may not remove all potential vulnerabilities.

2. Git history preservation

Git is designed to preserve a complete history of code changes. Even when the exposed secret is removed from the most recent commit, it might still exist in the commit history. Anyone with access to the repository can potentially access the sensitive information by examining older commits.

3. Forks and clones

Copies of the repository, such as forks or clones, might still contain the exposed secret. Even if the key is deleted from the original repository, it could persist in other copies.

4. Access control

In a collaborative development environment, ensuring all copies of the code are updated correctly is challenging. Other people might accidentally reintroduce the secret.

5. Risk of human error

Manual deletion of secrets can introduce the risk of human error. Developers might miss instances, forget to commit changes, or make mistakes during the removal process.

Key rotation is crucial 

Rotation invalidates the compromised key and replaces it with a new one. Even if the old key still sits in historical code or repositories, it no longer works.

Key rotation typically involves:

• generating a new key,
• updating the code and configurations to use the new key, and
• revoking or disabling the old key.

It shortens the time an attacker has to exploit a leaked secret. Here is what to plan for. 

Frequency

Rotate on a set schedule (e.g., every 30 days) or after specific events, such as a security breach. Your security requirements and policies set the pace.

Process

Generate a new key set (e.g., access key and secret key), point the API client at it, and revoke or disable the old keys once the new ones are active.

Compatibility

Clients should move to the new keys without service disruption. A grace period where both old and new keys are valid helps.

Monitoring

Watch rotation events and key usage. Unusual access patterns or authentication failures warrant a closer look.

Automation

Automate rotation with tools and scripts to cut human error.

Communication

Tell API consumers, such as developers or third-party applications: 

• when keys will change, and 
• how to update their configurations.

Compliance

Some industries and regulations require regular key rotation.

In short, rotation makes the leaked key useless, but you may still need a security audit or breach investigation to fully address the risks.

After key rotation: other remediation steps

Key rotation is not always enough. After a leak, you may also need these steps.

Audit and investigation

Find out how far the breach went and whether anyone used the leaked key maliciously. Check its usage logs for unauthorized access or data compromise from before the rotation.

Systems key rotation can’t reach

Rotation covers the systems under your direct control. Third-party services outside your purview that also use the key may need a notification or their own invalidation step.

Access controls after key rotation

Review access controls, permissions, and configurations, then adjust user privileges, restrict access, or add security measures as needed.

Data recovery

If someone accessed or compromised data with the leaked key, assess the loss, recover what you can, notify affected parties, and follow data protection regulations.

Prevention

Find the root cause of the leak and fix it: better key management, tighter access controls, and security training for staff.

In short, key rotation is one step. Dealing with the aftermath of leaked keys also takes auditing, investigation, and remediation.

Key rotation, in short

API key rotation involves periodically replacing authentication keys used to access an API. Deleting exposed secrets and cleaning up code history help after a leak, but key rotation is what makes leaked secrets ineffective, even if they persist in code history or repositories.