Let's talk about something embarrassing: hardcoded credentials.

You know that database password sitting in your config.js file? The API key in your environment variables? The AWS access keys committed to your Git repo (yes, I saw that)?

We've all done it. "I'll fix it later," you said. Then "later" became "eventually," which became "when we get hacked."

Here's the thing: managing secrets properly isn't hard anymore. AWS Secrets Manager handles encryption, rotation, versioning, and access control - all the stuff you should be doing but probably aren't.

Today we're fixing this. No more excuses. By the end of this tutorial, you'll know how to properly secure credentials for Lambda functions, EC2 instances, ECS tasks, and local development - using industry best practices.

Think of Secrets Manager like keeping your money in a bank instead of under your mattress. Sure, stuffing cash under your mattress is simpler. But when thieves come (and they will), you're cooked. The bank has vaults, guards, cameras, and insurance. That's Secrets Manager for your credentials.

Why Secrets Manager Instead of Environment Variables?

Environment variables:

  • Visible in process lists
  • Logged in CloudWatch Logs
  • Stored in plaintext in Lambda/ECS configurations
  • No automatic rotation
  • No audit trail
  • Exposed in stack traces and error messages

Secrets Manager:

  • Encrypted at rest and in transit
  • Access controlled via IAM policies
  • Automatic rotation (daily, weekly, custom)
  • Full audit trail (who accessed what, when)
  • Versioning (rollback to previous values)
  • Secret sharing across accounts/regions

What We're Building

A complete secrets management setup with:

  • Database credentials in Secrets Manager
  • Lambda function retrieving secrets
  • Automatic secret rotation
  • Local development workflow
  • Caching for performance
  • Error handling and monitoring

This guide assumes you're familiar with AWS Lambda functions - start there if you're new to serverless.

Prerequisites

  • AWS account
  • AWS CLI configured
  • Node.js 18+ (for Lambda examples)
  • Basic understanding of IAM

Step 1: Create Your First Secret

Using AWS CLI:

# Create database credentials secret
aws secretsmanager create-secret \
  --name prod/db/credentials \
  --description "Production database credentials" \
  --secret-string '{
    "username": "dbadmin",
    "password": "SuperSecure123!@#",
    "host": "prod-db.cluster-abc123.us-east-1.rds.amazonaws.com",
    "port": 5432,
    "database": "myapp"
  }' \
  --region us-east-1
# Create API key secret
aws secretsmanager create-secret \
  --name prod/api/openai \
  --secret-string '{
    "api_key": "sk-proj-aBcDeFgHiJkLmNoPqRsTuVwXyZ123456",
    "organization": "org-XYZ789"
  }' \
  --region us-east-1

Learn more about API key security when building with AI APIs in production environments. 

# Create OAuth credentials
aws secretsmanager create-secret \
  --name prod/oauth/github \
  --secret-string '{
    "client_id": "Iv1.a1b2c3d4e5f6g7h8",
    "client_secret": "a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0",
    "redirect_uri": "https://myapp.com/auth/callback"
  }' \
  --region us-east-1

Using AWS Console:

  1. Go to AWS Secrets Manager
  2. Click "Store a new secret"
  3. Choose secret type (Credentials for RDS database, Other type of secret, etc.)
  4. Enter key-value pairs or JSON
  5. Name your secret (use naming convention: {env}/{service}/{name})
  6. Configure rotation (optional)
  7. Review and store

Naming Conventions (Important): 

prod/db/credentials          # Production database
dev/db/credentials           # Development database
prod/api/stripe              # Stripe API keys
prod/auth/jwt-secret         # JWT signing secret
staging/cache/redis          # Redis connection string

Good naming makes secrets discoverable and prevents accidents (like using prod secrets in dev).

For Java/Spring Boot apps, integrate Secrets Manager with Spring Security JWT authentication for secure token signing.

Step 2: Retrieve Secrets in Lambda (Node.js)

Basic retrieval:

// lambda/index.js
const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager');

const client = new SecretsManagerClient({ region: 'us-east-1' });

/**
 * Retrieve secret from AWS Secrets Manager
 */
async function getSecret(secretName) {
  try {
    const response = await client.send(
      new GetSecretValueCommand({
        SecretId: secretName,
      })
    );

    // Parse the secret string
    return JSON.parse(response.SecretString);
  } catch (error) {
    console.error('Error retrieving secret:', error);
    throw error;
  }
}

// Usage
exports.handler = async (event) => {
  // Retrieve database credentials
  const dbCreds = await getSecret('prod/db/credentials');

  // Now you can use the credentials
  console.log(`Connecting to database at ${dbCreds.host}:${dbCreds.port}`);

  // Your database connection logic here
  // const db = new Database({
  //   host: dbCreds.host,
  //   port: dbCreds.port,
  //   user: dbCreds.username,
  //   password: dbCreds.password,
  //   database: dbCreds.database
  // });

  return {
    statusCode: 200,
    body: JSON.stringify({ message: 'Connected successfully' })
  };
};

Problem with this basic approach: Every Lambda invocation calls Secrets Manager. This adds latency (50-200ms) and costs money ($0.05 per 10,000 API calls).

Step 3: Add Caching (Production-Ready)

With in-memory caching:

// lambda/secrets.js
const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager');

const client = new SecretsManagerClient({ region: process.env.AWS_REGION || 'us-east-1' });

// In-memory cache
const secretCache = new Map();
const CACHE_TTL = 5 * 60 * 1000; // 5 minutes

/**
 * Get secret with caching
 */
async function getSecretCached(secretName) {
  const now = Date.now();
  const cached = secretCache.get(secretName);

  // Return cached value if still valid
  if (cached && (now - cached.timestamp) < CACHE_TTL) {
    console.log(`Cache hit for secret: ${secretName}`);
    return cached.value;
  }

  // Cache miss - fetch from Secrets Manager
  console.log(`Cache miss for secret: ${secretName} - fetching from Secrets Manager`);

  try {
    const response = await client.send(
      new GetSecretValueCommand({
        SecretId: secretName,
        VersionStage: 'AWSCURRENT', // Always get the current version
      })
    );

    const secretValue = JSON.parse(response.SecretString);

    // Update cache
    secretCache.set(secretName, {
      value: secretValue,
      timestamp: now,
    });

    return secretValue;
  } catch (error) {
    // If secret doesn't exist or access denied
    if (error.name === 'ResourceNotFoundException') {
      throw new Error(`Secret not found: ${secretName}`);
    }
    if (error.name === 'AccessDeniedException') {
      throw new Error(`Access denied to secret: ${secretName}`);
    }

    console.error('Error retrieving secret:', error);
    throw error;
  }
}

/**
 * Invalidate cache for a specific secret (useful after rotation)
 */
function invalidateCache(secretName) {
  secretCache.delete(secretName);
}

/**
 * Clear entire cache
 */
function clearCache() {
  secretCache.clear();
}

module.exports = {
  getSecretCached,
  invalidateCache,
  clearCache,
};

Using the cached version:

// lambda/index.js
const { getSecretCached } = require('./secrets');

exports.handler = async (event) => {
  try {
    // First call: fetches from Secrets Manager (~100ms)
    // Subsequent calls within 5 minutes: returns from cache (~1ms)
    const dbCreds = await getSecretCached('prod/db/credentials');
    const apiKeys = await getSecretCached('prod/api/openai');

    // Use the credentials
    // ... your business logic ...

    return {
      statusCode: 200,
      body: JSON.stringify({ message: 'Success' })
    };
  } catch (error) {
    console.error('Lambda error:', error);
    return {
      statusCode: 500,
      body: JSON.stringify({ error: error.message })
    };
  }
};

Apply this pattern in building serverless APIs with Lambda to securely manage API keys and database credentials.

Why caching matters:

  • Without cache: 100ms latency per secret fetch
  • With cache: 1ms latency for cached secrets
  • Cost savings: ~90% reduction in Secrets Manager API calls

Step 4: Configure IAM Permissions

Your Lambda function needs permission to access secrets:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowGetSecretValue",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/db/*",
        "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/api/*"
      ]
    },
    {
      "Sid": "AllowDecryptSecrets",
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt"
      ],
      "Resource": "arn:aws:kms:us-east-1:123456789012:key/your-kms-key-id"
    }
  ]
}

Attach to Lambda execution role:

aws iam put-role-policy \
  --role-name MyLambdaExecutionRole \
  --policy-name SecretsManagerAccess \
  --policy-document file://secrets-policy.json

Best practice: Use least privilege. Only grant access to specific secrets, not secretsmanager:* on all resources.

Step 5: Automatic Secret Rotation

Rotating secrets regularly reduces the blast radius if credentials leak.

For RDS databases (automatic):

# Enable automatic rotation for RDS credentials
aws secretsmanager rotate-secret \
  --secret-id prod/db/credentials \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789012:function:SecretsManagerRDSRotation \
  --rotation-rules AutomaticallyAfterDays=30

AWS provides pre-built rotation Lambda functions for RDS, Redshift, DocumentDB, and other AWS services.

For custom secrets (manual rotation function):

// rotation-lambda/index.js
const {
  SecretsManagerClient,
  GetSecretValueCommand,
  PutSecretValueCommand,
  UpdateSecretVersionStageCommand
} = require('@aws-sdk/client-secrets-manager');

const client = new SecretsManagerClient({ region: 'us-east-1' });

/**
 * Rotation Lambda handler
 * Called automatically by Secrets Manager
 */
exports.handler = async (event) => {
  const { SecretId, Token, Step } = event;

  console.log(`Rotating secret ${SecretId}, step: ${Step}`);

  switch (Step) {
    case 'createSecret':
      await createSecret(SecretId, Token);
      break;

    case 'setSecret':
      await setSecret(SecretId, Token);
      break;

    case 'testSecret':
      await testSecret(SecretId, Token);
      break;

    case 'finishSecret':
      await finishSecret(SecretId, Token);
      break;

    default:
      throw new Error(`Invalid step: ${Step}`);
  }
};

/**
 * Step 1: Create new secret version
 */
async function createSecret(secretId, token) {
  // Get current secret
  const current = await client.send(
    new GetSecretValueCommand({ SecretId: secretId })
  );

  const currentValue = JSON.parse(current.SecretString);

  // Generate new password (or fetch from external system)
  const newPassword = generateSecurePassword();

  const newValue = {
    ...currentValue,
    password: newPassword,
  };

  // Store new version with AWSPENDING label
  await client.send(
    new PutSecretValueCommand({
      SecretId: secretId,
      SecretString: JSON.stringify(newValue),
      VersionStages: ['AWSPENDING'],
      ClientRequestToken: token,
    })
  );
}

/**
 * Step 2: Update the actual service (database, API, etc.)
 */
async function setSecret(secretId, token) {
  // Get pending secret version
  const pending = await client.send(
    new GetSecretValueCommand({
      SecretId: secretId,
      VersionStage: 'AWSPENDING',
    })
  );

  const creds = JSON.parse(pending.SecretString);

  // Update database user password
  // This is where you'd connect to your database and change the password
  // await updateDatabasePassword(creds.username, creds.password);

  console.log('Secret updated in target service');
}

/**
 * Step 3: Test new secret works
 */
async function testSecret(secretId, token) {
  // Get pending secret
  const pending = await client.send(
    new GetSecretValueCommand({
      SecretId: secretId,
      VersionStage: 'AWSPENDING',
    })
  );

  const creds = JSON.parse(pending.SecretString);

  // Test connection with new credentials
  // try {
  //   await testDatabaseConnection(creds);
  //   console.log('New credentials work correctly');
  // } catch (error) {
  //   throw new Error('New credentials failed validation');
  // }
}

/**
 * Step 4: Promote new version to current
 */
async function finishSecret(secretId, token) {
  // Move AWSCURRENT label to new version
  await client.send(
    new UpdateSecretVersionStageCommand({
      SecretId: secretId,
      VersionStage: 'AWSCURRENT',
      MoveToVersionId: token,
      RemoveFromVersionId: await getCurrentVersionId(secretId),
    })
  );

  console.log('Secret rotation complete');
}

function generateSecurePassword() {
  // Use a proper password generation library in production
  const chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*';
  return Array.from({ length: 32 }, () => chars[Math.floor(Math.random() * chars.length)]).join('');
}

async function getCurrentVersionId(secretId) {
  const response = await client.send(
    new GetSecretValueCommand({
      SecretId: secretId,
      VersionStage: 'AWSCURRENT',
    })
  );
  return response.VersionId;
}

Enable rotation:

aws secretsmanager rotate-secret \
  --secret-id prod/api/custom \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789012:function:CustomSecretRotation \
  --rotation-rules AutomaticallyAfterDays=30

Step 6: Local Development Workflow

Don't hardcode secrets locally either. Use AWS CLI with profiles:

# ~/.aws/credentials
[dev]
aws_access_key_id = AKIA...
aws_secret_access_key = ...
region = us-east-1

[prod]
aws_access_key_id = AKIA...
aws_secret_access_key = ...
region = us-east-1

Local development script:

// local-dev.js
const { getSecretCached } = require('./secrets');

async function startApp() {
  // Use dev secrets locally
  const dbCreds = await getSecretCached('dev/db/credentials');

  console.log('Starting local development server...');
  console.log(`Database: ${dbCreds.host}`);

  // Start your app with the credentials
  // ...
}

startApp().catch(console.error);

Run with specific profile:

AWS_PROFILE=dev node local-dev.js

Never commit secrets to Git:

# .gitignore
.env
.env.local
secrets.json
credentials.json
*.pem
*.key

Step 7: Monitoring and Auditing

CloudWatch Alarms for suspicious activity:

# Alert on secret access failures
aws cloudwatch put-metric-alarm \
  --alarm-name SecretsManagerAccessDenied \
  --alarm-description "Alert when secrets access is denied" \
  --metric-name UserErrorCount \
  --namespace AWS/SecretsManager \
  --statistic Sum \
  --period 300 \
  --threshold 5 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1

CloudTrail logging:

All Secrets Manager API calls are logged in CloudTrail. Monitor for:

  • Unexpected secret retrievals
  • Secret modifications
  • Failed access attempts
  • Secrets accessed from unknown IPs

Query CloudTrail logs:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=prod/db/credentials \
  --max-results 50

Best Practices Checklist

  • [x] Use Secrets Manager for all credentials - No hardcoded passwords
  • [x] Implement caching - 5-minute TTL balances security and performance
  • [x] Least privilege IAM policies - Grant access to specific secrets only
  • [x] Enable rotation - 30-90 days for most secrets
  • [x] Use naming conventions - {env}/{service}/{name}
  • [x] Monitor access - CloudWatch alarms and CloudTrail logs
  • [x] Separate dev/staging/prod - Never use prod secrets in non-prod environments
  • [x] Version secrets - Keep old versions for emergency rollback
  • [x] Encrypt with KMS - Use customer-managed keys for sensitive secrets
  • [x] Test rotation - Regularly test your rotation logic

Common Mistakes to Avoid

1. Logging secrets: 

// ❌ DON'T
console.log('Database password:', dbCreds.password);

// ✅ DO
console.log('Database connection successful');

2. Long cache TTL: 

// ❌ DON'T - 24 hour cache
const CACHE_TTL = 24 * 60 * 60 * 1000;

// ✅ DO - 5 minute cache
const CACHE_TTL = 5 * 60 * 1000;

3. Overly permissive IAM: 

// ❌ DON'T
"Resource": "*"

// ✅ DO
"Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/api/*"

4. No error handling: 

// ❌ DON'T
const secret = await getSecret('prod/db/credentials');

// ✅ DO
try {
  const secret = await getSecret('prod/db/credentials');
} catch (error) {
  console.error('Failed to retrieve secret:', error.message);
  throw new Error('Unable to connect to database');
}

Cost Breakdown (2025 Pricing)

Secrets Manager costs:

  • $0.40 per secret per month
  • $0.05 per 10,000 API calls

Example: 10 secrets, 1M API calls/month

  • Secrets: 10 × $0.40 = $4.00
  • API calls: 1M / 10,000 × $0.05 = $5.00
  • Total: $9.00/month

With caching (90% reduction):

  • Secrets: $4.00
  • API calls: 100K / 10,000 × $0.05 = $0.50
  • Total: $4.50/month

Caching saves ~50% on Secrets Manager costs.

The Bottom Line

Stop treating credentials like they're not valuable. That database password? It's the keys to your kingdom.

Secrets Manager makes proper secret management easy:

  • Encrypted storage
  • Automatic rotation
  • Access control
  • Full audit trail
  • Versioning

Combined with caching, the performance impact is negligible and the cost is pennies. YesYour future self (and your security team) will thank you for implementing this properly from day one.

Now go rotate those secrets.

Sources: