Olivia A. Gallucci

This blog post explores how key rotation enhances the security of application programming interfaces (APIs)—emphasizing the necessity of key rotation alongside the conventional practice of identifying and deleting leaked secret keys—to mitigate security risks and ensure the long-term integrity of digital systems.

Overview 

APIs, keys, and rotations belong to the broader concept of computer programming and software development. These terms describe various aspects and components of software systems in computer programming and software development.

APIs

APIs are sets of rules and protocols that allow software applications to communicate and interact with each other, enabling them to share data and functionality, and facilitating the integration of diverse systems and services.

For example, somebody could use APIs to learn about a GitHub profile.

# Use 'pip install requests' before running. 
import requests

# replace 'YOUR_GITHUB_USERNAME' with your GitHub username
github_username = 'YOUR_GITHUB_USERNAME'

# github API endpoint to retrieve user information
api_url = f'https://api.github.com/users/{github_username}'

try:
    # send a GET request to the GitHub API
    response = requests.get(api_url)

    # check if the request was successful (status code 200)
    if response.status_code == 200:
        # parse the JSON response
        user_data = response.json()

        # print some user information
        print(f"github username: {user_data['login']}")
        print(f"name: {user_data['name']}")
        print(f"bio: {user_data['bio']}")
        print(f"followers: {user_data['followers']}")
        print(f"following: {user_data['following']}")
        print(f"public repositories: {user_data['public_repos']}")
    else:
        # print an error message if the request was not successful
        print(f"request failed with status code {response.status_code}")
except requests.exceptions.RequestException as e:
    # handle any exceptions that may occur during the request
    print(f"an error occurred: {e}")

Gist

Other examples 

However, there are tons of other uses for APIs. 

Problems with APIs

Although the public often utilizes APIs, some APIs are for a specific group of people. One example of this would be an API for a banking application where customers need to access their account balances and make transactions securely.

Unfortunately, malicious actors sometimes obtain unauthenticated or unauthorized access to APIs, leading to data breaches, data manipulation, denial-of-service attacks, and SQL or code injection attacks.

Authentication

Authentication is an act, process, or method of showing something (such as an identity, a piece of art, or a financial transaction) to be real, true, or genuine : the act or process of authenticating something.

Merriam-Webster 

According to Fortinet, “the user must present physical or nonphysical evidence (information) to the authentication platform to confirm the user’s identity.” Fortinet divides these confirmation practices into three categories: 

• What they have: The possession of a physical object, such as a key, keycard, key fob, or swipe card.
• What they know: Information that only the user would know, including a password, passcode, personal identification number (PIN), date of birth, Social Security number, or other personally identifiable information (PII).
• Who they are: Biometrics, or the use of an index finger, thumb, hand, voice, retina, face, or another unique physical identifier to gain access to a resource. The physical attribute must match what was used at the time of the user’s enrollment in the system.

For this blog post, the main method of authentication referenced is “what [the user] knows” (e.g., password-based authentication). With this form of authentication, an attacker could exploit an organization’s weak password policies by using brute-force or credential stuffing to access a user’s account.

Read more: CIA Triad: Confidentiality, Integrity, Availability. 

Authorization

Authorization “expresses permissions and rights granted to a user” (TrendMicro). It answers the question, “what is a [user/system/group] allowed to do?”

For instance, a manager granting an employee permission to access confidential company financial data, while denying access to other employees.

When authorization issues are present, attackers can perform actions they should not be able to. For instance, once inside a user’s account through legitimate authentication, an attacker might manipulate the API request to access another user’s account or attempt to execute transactions without proper authorization, exploiting vulnerabilities in the authorization process.

Keys

To combat unauthenticated and unauthorized access to data and services, APIs often require keys—sometimes called “API keys” or “authentication tokens.”

Each key is a piece of information, typically a string of characters or bits, used to encrypt, decrypt, or authenticate data in various cryptographic systems, ensuring its security and integrity.

Authentication code

Here is an example HTTP request with an “Authorization” header for basic authentication. The “Authorization” header contains the word “Basic” followed by a base64-encoded username and password. This is a basic authentication scheme where the server can decode the credentials to authenticate the client.

GET /api/resource HTTP/1.1
Host: example.com
Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==

Gist

Below is an example of a full HTTP request with an Authorization header containing an API key or access token.

GET /api/resource HTTP/1.1
Host: example.com
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

Gist

• GET /api/resource is the requested HTTP method and the endpoint.
• Host: example.com specifies the hostname of the server.
• Authorization: Bearer … is the Authorization header containing the API key or access token

Here, keys ensure that only authorized users or applications can access their services and data. Keys also allow for tracking usage for security and billing purposes.

The table below details how APIs commonly use keys. 

Common API key usage 

Here, APIs use keys or tokens for authentication and authorization, allowing them to verify the identity of users or applications and control their access to specific resources and functionalities. The choice of key or token mechanism depends on the API’s security requirements and the complexity of the authorization process.

Attack and remediation examples 

Without keys or proper authentication, APIs are vulnerable to unauthorized access, abuse, data breaches, and potential overuse of resources, leading to security risks and service disruptions.

Several types of attacks can occur when APIs do not use proper authentication mechanisms such as API keys, including:

Implementing authentication mechanisms like API keys, OAuth tokens, or other access control methods is essential to mitigate these potential risks and secure APIs.

Leaky keys

As we discussed, API keys are critical pieces of authentication and authorization used to access and secure various online services and resources. Thus, exposed API keys pose significant security risks. Below, I listed some common mistakes that lead to API key leaks. 

• Storing keys in code repositories
  • Developers often mistakenly commit API keys directly into source code repositories, where they can be easily accessed by anyone with access to the repository’s history.
• Poor access control
  • Failing to restrict access to API keys or granting excessive permissions can result in unintended users gaining access to these keys.
• Lack of encryption
  • Transmitting or storing API keys without encryption makes them susceptible to interception by attackers during data transmission or storage.
• Hardcoding keys
  • Embedding API keys directly into applications or scripts without utilizing secure storage methods can lead to inadvertent exposure.
• Inadequate monitoring and auditing
  • Failing to monitor API key usage and unusual activity can delay the detection of breaches or unauthorized access.

Keys scanning

Unfortunately, developers frequently leak API keys, so multiple products and services developed strategies to help mitigate these risks. One of these tools is TruffleHog, an open-source security tool designed to search through the commit history of Git repositories for high-entropy strings, which are often indicative of sensitive information like API keys, passwords, and other secrets.

TruffleHog

TruffleHog is primarily used for identifying and mitigating security risks related to code repositories. It works by scanning the commit history of a Git repository and analyzing the contents of each commit to detect strings with high entropy, which suggests that they might be sensitive data.

A string with high entropy contains a seemingly random arrangement of characters, making it difficult to predict or compress. High-entropy strings can be detected by their lack of discernible patterns or repetitions, like  “R#t$5&@pL!9KzWq2*X.” 

Here is an example of a high entropy string found by TruffleHog (specifically, the raw result value “AKIAYVP4CIPPERUVIFXG”): 

$ trufflehog git https://github.com/trufflesecurity/test_keys --only-verified
🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

Found verified result 🐷🔑
Detector Type: AWS
Decoder Type: PLAIN
Raw result: AKIAYVP4CIPPERUVIFXG
Line: 4
Commit: fbc14303ffbf8fb1c2c1914e8dda7d0121633aca
File: keys
Email: counter <counter@counters-MacBook-Air.local>
Repository: https://github.com/trufflesecurity/test_keys
Timestamp: 2022-06-16 10:17:40 -0700 PDT
…

Commit Source

By running TruffleHog on a codebase, developers and security engineers can proactively identify and remove exposed secrets from repositories, helping to prevent breaches and unauthorized access. It is a valuable tool for maintaining the security of code repositories and ensuring that sensitive information is not inadvertently leaked or exposed to potential attackers.

Why API keys and TruffleHog? 

My friend contacted me and asked if I would be interested in contributing to How To Rotate  (HTR), an open-source collection of API key rotation tutorials by Truffle Security. 

My friend sent me two links on how I could start: 

• How to Rotate Leaked API Keys by Truffle Security 
• How to Rotate: Key Rotation Tutorials by Joe Leon

Since I do not know much about APIs or key rotations, contributing to HTR would be a great way to help other beginners contribute to an open-source project through technical, step-by-step blog posts since the blogs come from a beginner mindset and provide details that an expert might overlook.

HTR maintainers will review all of my contributions to ensure they are accurate and provide value to the HTR community.

Outside of TruffleHog, other tools help detect high-entropy strings, like Gitrob (archived), Gitleaks (creator works for Truffle Security), and GitGuardian (commercial tool). I am focusing on TruffleHog because I understand the importance of supportive and dedicated communities when reaching out for maintenance and relying on others for long-term growth and support. Truffle Security’s dedication to open-source software, collaboration, and transparency shows its focus on advancing the computing and security communities. 

In addition, it makes me happy to contribute to a project my friends made.

Scanning limitations and remediation attempts 

Despite the benefits of using scanners like TruffleHog, there are limitations to these tools; they don’t prevent key usage. Thus, the keys found by scanners could still work and be an active threat to systems and organizations. 

Once organizations find leaked keys, they often take immediate action to revoke or invalidate them. However, revoking and invalidating keys is often tedious, time-consuming, and error-filled. Now, let’s explore how to revoke and invalidate keys.

Revoke vs invalidate 

“Revoke keys” and “invalidate keys” are related concepts but can have slightly different implications depending on the context. Both terms generally refer to rendering cryptographic keys or access credentials unusable for security reasons. However, the choice of terminology can reflect different aspects of the process. 

Revocation

Revoking keys usually implies taking specific action to cancel or withdraw the validity of keys. It often suggests a deliberate and controlled process where an authority, such as a system administrator or a security team, actively revokes or deactivates keys. Lastly, revoking keys may involve explicit steps to inform systems and users that the keys are no longer valid.

Organizations typically revoke keys by disabling or deactivating them in the systems or services where they are used and generating new keys to replace the compromised ones.

Invalidation

Invalidating keys generally means making keys or credentials invalid or unusable. The term “invalidate” may be more general and can encompass how keys become unusable, including revocation. Invalidation can result from key expiration, password changes, or security breaches.

Organizations typically invalidate keys by ensuring they are not recognized as valid credentials within the systems or services they were originally used for, making them useless for authentication or access.

In practice, “revoking keys” and “invalidating keys” are often used interchangeably. However, I wanted to explain the difference since this post is about keys. “Revoke keys” and “invalidate keys” convey that the keys are no longer functional or trusted for authentication or encryption purposes.

Pitfalls of invalidation and revocation

Invalidating and revoking API keys can disrupt legitimate services and cause downtime for applications, potentially impacting users and revenue. 

As a result, key rotation is often used instead of or in addition to revoking and invalidating keys. Rotating keys involves replacing the old key with a new one, allowing existing systems to continue functioning with minimal disruption and no need for immediate manual reconfiguration.

Key rotation

Key rotation is the process of updating and replacing cryptographic keys to enhance security and minimize the risk associated with compromised or outdated keys.

In Leon’s blog post, he mentions that: 

Security engineers and developers run our open-source secret scanner, TruffleHog, to find and remediate leaked secrets across their Software Development Life Cycle. While TruffleHog can detect over 750 types of leaked secrets, that only solves part of the problem. Users often seek our advice on remediation.

With very, very few exceptions, the most effective way to remediate a leaked secret is with key rotation. Simply deleting the code that exposes the key or editing the Git history is insufficient.

But we are left wondering why “deleting the code that exposes the key or editing Git history is insufficient.” 

The big question is: 

Outside of the pitfalls of invalidating and revoking keys, why is key rotation needed? 

Why key rotation? 

Deleting the code that exposes the key or editing Git history, while useful steps in response to a leaked secret, are often insufficient on their own for several reasons. 

1. Incomplete removal

When a secret is exposed in code, it may not be limited to a single file or commit. Secrets can be spread across multiple code files, configuration files, or external services. Deleting one instance of the exposed key may not remove all potential vulnerabilities.

2. Git history preservation

Git is designed to preserve a complete history of code changes. Even when the exposed secret is removed from the most recent commit, it might still exist in the commit history. Anyone with access to the repository can potentially access the sensitive information by examining older commits.

3. Forks and clones

Copies of the repository, such as forks or clones, might still contain the exposed secret. Even if the key is deleted from the original repository, it could persist in other copies.

4. Access control

In a collaborative development environment, ensuring all copies of the code are updated correctly is challenging. Other people might accidentally reintroduce the secret.

5. Risk of human error

Manual deletion of secrets can introduce the risk of human error. Developers might miss instances, forget to commit changes, or make mistakes during the removal process.

Key rotation is crucial 

These concerns are where key rotation becomes crucial. 

By rotating keys, engineers invalidate the compromised key and replace it with a new one. Rotating keys ensures that even if the old key is still present in historical code or repositories, it is no longer valid or usable. Key rotation is a proactive measure that helps prevent unauthorized access to sensitive resources or data.

Key rotation typically involves:

• generating a new key,
• updating the code and configurations to use the new key, and
• revoking or disabling the old key.

It reduces the window of opportunity for malicious actors who may have obtained the leaked secret to exploit it. Below, I listed some of the notable aspects of API key rotation. 

Frequency

The frequency of key rotation can vary depending on security requirements and policies. It can be done on a set schedule (e.g., every 30 days) or in response to specific events, such as a security breach.

Process

The key rotation process typically involves generating a new set of keys (e.g., access key and secret key) and updating the API client to use these new keys. Once the new keys are active, the old keys are revoked or disabled.

Compatibility

During key rotation, ensuring existing API clients can seamlessly transition to the new keys without service disruption is ideal. This may involve providing a grace period during which both the old and new keys are valid.

Monitoring

Organizations often monitor key rotation events and key usage to detect suspicious activities. Unusual patterns of access or authentication failures can trigger further investigation.

Automation

In software development and operations, engineers often automate key rotation with tools and scripts to reduce the likelihood of human error.

Communication

Clear communication is crucial during key rotation. API consumers, such as developers or third-party applications, should be: 

• notified of upcoming key changes and 
• provided with instructions on updating their configurations.

Compliance

In some industries and regulatory environments, regular key rotation may be a compliance requirement to ensure data security and privacy.

In short, key rotation can help address both problems of leaked keys by rendering the compromised keys useless and replacing them with new, secure keys, effectively revoking and invalidating the leaked keys in the process. However, organizations may still need to take additional actions, such as conducting security audits and investigating potential breaches, to ensure comprehensive mitigation of the risks associated with the leaked keys.

Other necessary remediation steps 

While key rotation is a crucial step in mitigating the risks associated with leaked keys, organizations may still need to take additional actions. Below, I listed some reasons why organizations may benefit from taking additional steps.

Audit and investigation

It is essential to understand the extent of the potential breach and determine whether the leaked keys were used maliciously. Conducting a security audit and investigation helps identify any unauthorized access or data compromise that may have occurred before the key rotation.

Affected systems

Various systems and services could use leaked keys. Key rotation typically addresses the systems under the organization’s direct control. However, there may be third-party services or systems outside of the organization’s purview that also use the keys. These systems may require notification or actions to invalidate the leaked keys.

Access controls

Organizations should review access controls, permissions, and configurations to ensure that unauthorized access is prevented even after key rotation. This process may involve adjusting user privileges, restricting access, or implementing additional security measures.

Data recovery

If leaked keys resulted in accessed or compromised data, organizations may need to assess and address data loss or potential data breaches. This process could involve data recovery efforts, notifying affected parties, and complying with data protection regulations.

Prevention

To prevent similar incidents, organizations should identify the root cause of the key leak and implement measures to strengthen security practices, including improved key management, access controls, and security training for staff.

In short, key rotation is a critical step. However, addressing the aftermath of leaked keys often requires a comprehensive approach that includes auditing, investigation, and remediation efforts to ensure the security and integrity of systems and data.

Conclusion 

API key rotation involves periodically replacing authentication keys used to access an API. While deleting exposed secrets and cleaning up code history are essential steps in responding to a security breach, key rotation is a critical component of the remediation process. It helps mitigate the risks of leaked secrets by rendering them ineffective, even if they persist in code history or repositories.