Secrets Management

The Hotel Master Keycard

Imagine checking into a modern hotel. The receptionist doesn't hand you a metal key that can open every room in the building. Instead, they encode a plastic keycard that only opens your specific room, and only for the next 48 hours. When you check out, the card deactivates.

If you lose the keycard on the street, the hotel doesn't have to change the locks on the doors; they just instantly revoke that specific card's access in their computer system.

In system design, our applications constantly need to access private resources: databases, third-party APIs (like Stripe or Twilio), and internal microservices. To do this, they need "keys" in the form of passwords, API tokens, and TLS certificates.

Secrets Management is the system we use to securely store, distribute, and track these digital keys, ensuring that if a key gets lost or stolen, it can't be used to compromise the entire building.

The "Old Way" (How to get hacked)

Before dedicated secrets management tools existed, developers did terrible things with secrets:

1. Hardcoding: They pasted the database password directly into the source code. When the code was pushed to GitHub, anyone with access to the repository could see the password. If the repo was public, bots would scrape the password in seconds.
2. Environment Variables (.env files): A step up, but teams often emailed .env files to each other or left them sitting plainly on servers.
3. Configuration Files: Storing API keys in plain text JSON files on the server disk.

If a hacker managed to exploit a minor bug in the web app and read a file off the server (a path traversal attack), they would instantly find the plain text database password, the AWS keys, and the Stripe API key. Game over.

How a Secrets Manager Works

A Secrets Manager (like HashiCorp Vault, AWS Secrets Manager, or Google Secret Manager) acts as an ultra-secure, centralized digital vault for your infrastructure.

Instead of writing a database password in a configuration file, the application is programmed to ask the Secrets Manager for the password right as it boots up.

The Workflow

Let's look at how a modern microservice interacts with a Secrets Manager:

1. Authentication: The application boots up and proves its identity to the Secrets Manager (e.g., "I am the Payment Service running on AWS Node 5").
2. Authorization: The Secrets Manager checks its policies. ("Does the Payment Service have permission to see the Stripe API key? Yes.")
3. Retrieval: The Secrets Manager securely delivers the secret to the application's in-memory RAM.

Notice that the secret is never written to disk, and it's never visible in the source code.

Press enter or space to select a node. You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.

Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.

algobase.dev

Secrets manager workflow: the service authenticates with Vault on startup, Vault checks its access policy, then issues a credential directly into the service's memory. The credential is never written to disk or source code. When the credential expires or the engineer revokes it, the service loses database access immediately.

1 / 10.5×1×2×3×Playground

Secrets manager workflow: service authenticates with Vault, receives a credential in memory, and uses it to connect to the database

The Magic of Dynamic Secrets

The most powerful feature of advanced tools like HashiCorp Vault is the concept of Dynamic Secrets.

Let's say your web server needs to query a PostgreSQL database. Traditionally, you create a database user called web_app with a static password, and you put that password in the Secrets Manager.

But what if we could eliminate static passwords entirely?

With Dynamic Secrets:

1. The web app asks Vault for database access.
2. Vault dynamically connects to PostgreSQL and creates a brand new database user and password on the fly.
3. Vault hands these temporary credentials to the web app.
4. Vault sets a "Time to Live" (TTL). After 1 hour, Vault automatically logs into PostgreSQL and deletes that user account.

If a hacker somehow steals the password out of the server's RAM, the password is only valid for a few minutes. By the time the hacker tries to use it, the password has already self-destructed. The blast radius of a leak is reduced to near zero.

Rotation and Revocation

Secrets Managers also solve the headache of Rotation. Security best practices dictate that you should change your API keys and passwords every 30 to 90 days. If you hardcode passwords, this requires massive coordination and downtime to deploy new code.

With a Secrets Manager, an administrator can simply log in, update the value of the Stripe API key, and all the microservices will instantly start using the new key without any code changes.

Furthermore, if an engineer's laptop is stolen, administrators can click a single button to Revoke any secrets associated with that engineer, instantly neutralizing the threat.

Summary

• Secrets Management is the practice of securely storing and distributing sensitive data like API keys, database passwords, and certificates.
• Never hardcode secrets in source code or commit them to version control.
• A Secrets Manager acts as a centralized vault. Applications authenticate with the vault to retrieve secrets directly into memory.
• Advanced platforms offer Dynamic Secrets, creating temporary, self-destructing credentials on demand, drastically reducing the impact of leaked keys.
• Centralized systems make it trivial to rotate (change) and revoke (cancel) compromised credentials without redeploying code.