Start a project
Advanced

The Secret Was 'Secured' — Until We Found the Default Password

6 min read
Project:Agency OS
Stack:
Node.jsTypeScriptNext.jsSecurityDevSecOps

We engineered a robust internal API authentication system, only to discover during a red team audit that a single hardcoded default string silently bypassed the entire security boundary. Here is why applications must fail closed.

Security engineering is a constant battle against convenience. When building distributed systems, we often introduce shortcuts to make local development faster for the engineering team. But if those shortcuts are allowed to leak into production environments, they transform from conveniences into catastrophic vulnerabilities.
This is the story of how we built a highly secure, zero-trust internal API architecture for Agency OS, only to watch the entire security boundary collapse during a red team audit—not because of a zero-day exploit, but because of a single line of JavaScript written to make our lives easier.
---

1. The Architecture: Securing the Internal Boundary

Agency OS operates as a decoupled monorepo. We have a central dashboard application that acts as the control plane, and multiple client storefront applications that consume data from it.
To prevent unauthorized access, we engineered a strict internal authentication system. The storefront applications were required to pass a cryptographic header (x-internal-secret) when making cross-service network requests to the dashboard. If the header was missing or incorrect, the dashboard would aggressively reject the request with a 401 Unauthorized.
The logical flow looked rock solid:
  1. Storefront initiates a fetch request.
  2. Appends x-internal-secret header.
  3. Dashboard receives the request.
  4. Validates the header against process.env.INTERNAL_API_SECRET.
  5. If valid, returns sensitive client data.
  6. If invalid, blocks access.
We deployed the system. It worked flawlessly. The perimeter was secured. We patted ourselves on the back.
---

2. The Red Team Audit: Round 2

A few weeks later, we initiated our scheduled "Round 2" internal red team audit. The goal was to stress-test the microservice communication layer we had just built.
I handed the repository over to the security engineers. I expected them to try SQL injection, JWT manipulation, or complex Server-Side Request Forgery (SSRF) attacks.
Instead, ten minutes into the audit, I received a Slack message containing a single Proof-of-Concept curl request.
# The Red Team Payload
GET [https://dashboard.agency.com/api/clients/client_002/secrets](https://dashboard.agency.com/api/clients/client_002/secrets)
x-internal-secret: agency_os_internal_secret_default
When I ran the curl command against our staging environment, the server bypassed all authorization checks, opened the vault, and returned the highly sensitive Shopify access token for client_002. No user authentication. No JWTs. No IP whitelisting.
The entire internal authentication model had collapsed to a single, known constant string.
---

3. The Flaw: The Convenience Shortcut

I immediately pulled up the codebase and ran a global grep for the string agency_os_internal_secret_default. The terminal output was devastating. The string was littered across 8 different critical files in the monorepo:
1// apps/dashboard/proxy.ts:6
2// apps/dashboard/app/api/clients/[storeId]/secrets/route.ts:4
3// apps/dashboard/app/api/reviews/route.ts:6
4// apps/storefront/app/lib/tenantShopify.ts:8
5// packages/config/registry.ts:12
6// ... and 3 more
7
8const INTERNAL_SECRET = process.env.INTERNAL_API_SECRET || 'agency_os_internal_secret_default';
9
When we first built the internal API layer, the developers found it annoying that their local npm run dev servers kept crashing because they hadn't set up their .env.local files yet.
To "fix" the developer experience, someone added the JavaScript OR (||) operator. The logic was: If the environment variable exists, use it. If not, just use this hardcoded default string so the local server doesn't crash.

The Two-Part Disaster

The vulnerability was a confluence of two failures:
Failure 1: The Missing Production Configuration
When the infrastructure team deployed the new architecture to the Vercel staging and production environments, they forgot to actually provision the INTERNAL_API_SECRET environment variable in the cloud dashboard.
Failure 2: The Silent Downgrade
Because the variable was missing, the Node.js process evaluated process.env.INTERNAL_API_SECRET as undefined. Instead of throwing a fatal error and refusing to boot, the application silently downgraded to the right side of the || operator. It defaulted the master key for the entire platform to 'agency_os_internal_secret_default'.
Because this string was hardcoded directly in the TypeScript source code, it existed in the Git history. Anyone with access to the source code (or anyone who could guess a highly predictable default naming convention) immediately knew the master password to the production database.
---

4. Engineering Principle: Systems Must Fail Closed

This incident highlighted a fundamental flaw in our engineering culture. We had prioritized local developer convenience over production security principles.
In security engineering, systems must always fail closed.
If a system lacks the necessary cryptographic configurations, authentication keys, or permissions to operate securely, it must refuse to operate at all. A crashed server is an operational incident (which gets fixed in five minutes). A server that silently downgrades its security to stay alive is a data breach (which destroys companies).
Our Node.js application was explicitly designed to "fail open" by providing a fallback string. The system chose to stay alive at the cost of its integrity.
---

5. The Fix: Enforcing the Boundary

The remediation was straightforward but structurally critical. We stripped every instance of the || fallback operator out of the codebase.
Instead of relying on decentralized, inline evaluations scattered across 8 different files, we centralized the secret retrieval into a canonical helper utility:
1// packages/config/secrets.ts
2
3export function getInternalSecret(): string {
4  const secret = process.env.INTERNAL_API_SECRET;
5  
6  // In production, if the secret is missing, return an empty string
7  // This ensures no static bypass token will ever mathematically match
8  if (!secret) {
9    if (process.env.NODE_ENV === 'production') {
10       // Optional: Log a critical alert to Datadog/Sentry here
11       return ""; 
12    }
13    // Only allow fallback in local dev environments
14    return "agency_os_internal_secret_default";
15  }
16  
17  return secret;
18}
19
By enforcing this canonical helper, if the infrastructure team forgets to provision the secret in a production environment, getInternalSecret() returns an empty string ("").
When an attacker (or a legitimate internal service) sends a request with the header x-internal-secret: "agency_os_internal_secret_default", the server compares it to the empty string "". The comparison fails, and the server returns a 401 Unauthorized.
The system now fails securely.
---

6. The Final Evolution: Strict Startup Validation

While the helper function solved the immediate vulnerability, the true architectural solution is to prevent the application from even booting if it is misconfigured.
In our modern iteration of Agency OS, we rely on Zod (a TypeScript schema validation library) to validate process.env at startup.
1// env.ts
2import { z } from 'zod';
3
4const envSchema = z.object({
5  DATABASE_URL: z.string().url(),
6  INTERNAL_API_SECRET: z.string().min(32), // Must be provided and cryptographically long
7});
8
9// This will physically crash the Node process if variables are missing
10export const env = envSchema.parse(process.env); 
11
Now, if a developer forgets to add the secret to Vercel, the build pipeline violently crashes with a ZodError before the code ever reaches the edge network. We have entirely eliminated the possibility of a silent downgrade.
---

Conclusion

The most terrifying bugs are the ones that don't make any noise.
When a database connection drops, logs fill up with stack traces, alerts trigger, and engineers immediately know something is wrong. But when an application silently falls back to a hardcoded default string, the server stays green. The application feels fast. The metrics look healthy.
You will not know you have a catastrophic vulnerability until a red team auditor—or a malicious actor—points it out.
Never use the || operator to provide fallback security credentials in your source code. Developer convenience on localhost is never worth compromising the integrity of your production environment. If a secret is missing, let the application crash. It is the safest thing it can possibly do.
---
---

💡 Key Engineering Takeaways

Security mechanisms must always 'fail closed'; if a required configuration is missing, the application should crash, not silently downgrade. Hardcoding fallback secrets in source code completely invalidates the security boundary because version control history is rarely truly secret. Environment variables (like INTERNAL_API_SECRET) must be strictly enforced at startup using validation schemas (like Zod) to prevent silent defaults.

Frequently Asked Questions

What is a hardcoded default secret vulnerability?

This vulnerability occurs when developers hardcode a fallback password or API key directly into the application's source code (e.g., `process.env.SECRET || "default_password"`). If the production environment is misconfigured and the environment variable is missing, the application silently relies on the publicly visible, insecure default string.

Why is "failing closed" a critical security principle?

"Failing closed" means that if a system encounters an error, missing configuration, or indeterminate state, it defaults to rejecting access or shutting down. "Failing open" (like using a fallback password to keep the server running) prioritizes uptime over security, which often leads to catastrophic data breaches.

Why shouldn't default secrets be stored in version control (Git)?

Version control history is permanent and often accessible by dozens of developers, contractors, or third-party CI/CD tools. Storing any valid secret or fallback password in plaintext within Git means that if the repository is ever compromised or leaked, the attackers instantly have the keys to your production systems.

How does strict startup validation (like Zod) prevent this issue?

Startup validation parses the `process.env` object the moment the Node.js server boots up. If critical security variables (like `INTERNAL_API_SECRET`) are missing or do not meet length requirements, the validation library intentionally crashes the application, preventing it from serving traffic in an insecure, unconfigured state.

How do I fix hardcoded secrets in my existing codebase?

First, audit the codebase using grep for operators like `|| 'secret'` attached to `process.env`. Second, centralize all environment variable access into a single configuration file. Third, ensure that if a variable is missing in production, the application either throws a fatal error or returns a secure, unmatchable value (like an empty string).

Feedback

Was this article helpful?

Related Engineering Diaries