Skip to main content
MK
Mohan Kumar BN Platform Engineer · Kubestronaut
Security & Supply Chain Enterprise OpenShift & Hybrid GKE/On-Premises Kubernetes Fleets

Zero-Trust Secrets Management & Software Supply Chain Security

Eliminating static credentials with HashiCorp Vault & ESO while enforcing Cosign image signing, OPA Gatekeeper, and Wiz CNAPP

ROLE & OWNERSHIP

Senior Platform & Security Engineer

Zero Hard-Coded Secrets

Eliminated static application credentials across multi-cluster OpenShift and Kubernetes workloads via Vault + ESO.

100% Signed Images

Enforced cryptographic image provenance and CVE gating using Cosign, Trivy, Snyk, and OPA Policy Agent.

Continuous Posture

Integrated Wiz CNAPP and enterprise OAuth/RBAC to close posture gaps proactively.

01 // Engineering Problem

Hard-coded secrets in CI variables, unsigned container images entering clusters, and fragmented visibility into runtime security posture created critical compliance and supply chain risks.

02 // Business & Operational Context

Across both enterprise OpenShift platforms at Henkel and SaaS Kubernetes fleets at OpsRamp, security compliance required eliminating static credentials, cryptographically verifying every container image before execution, and continuously auditing cluster posture.

Architecture Topology // Zero-Trust Secrets Management & Software Supply Chain Security

TOPOLOGY // DECLARATIVE BLUEPRINT
01 // Vulnerability Gate 01

Trivy + Snyk + Harbor

Automated CVE scanning in CI and inside the self-hosted Harbor OCI registry before artifacts can be promoted.

02 // Provenance 02

Sigstore Cosign + OPA

Cryptographic signing of container digests with Cosign and cluster admission enforcement via OPA Policy Agent.

03 // Secret Delivery 03

HashiCorp Vault + ESO

Workload identity JWT authentication from Kubernetes ServiceAccounts to Vault via External Secrets Operator.

04 // Runtime Posture 04

Wiz CNAPP + OpenShift SCCs

Continuous cloud-native posture management, RBAC least-privilege auditing, and non-root execution guardrails.

Mermaid Topology Specification (Architecture-as-Code) ▼
flowchart LR
  subgraph SupplyChain["Secure Software Supply Chain"]
    CI["CI Build Pipeline\n(Tekton / GitLab / Jenkins)"] --> Scan["CVE Vulnerability Gate\n(Trivy + Snyk)"]
    Scan --> Harbor["Harbor Enterprise Registry\n(Signed OCI Artifacts)"]
    Scan --> Cosign["Sigstore Cosign\n(Cryptographic Signature)"]
  end
  subgraph Runtime["Zero-Trust Kubernetes / OpenShift Runtime"]
    OPA["OPA Gatekeeper Admission\n(Verify Signature + Policy)"]
    Vault["HashiCorp Vault\n(Central Secret Engine)"]
    ESO["External Secrets Operator\n(ClusterSecretStore + CRD)"]
    Pod["Hardened Application Pod\n(Non-Root SCC + Runtime Secrets)"]
    Wiz["Wiz CNAPP\n(Continuous Posture & Risk Scan)"]
  end
  Harbor --> OPA
  Cosign --> OPA
  OPA -->|"Admit Verified Image"| Pod
  Vault <-->|"K8s JWT Auth"| ESO
  ESO -->|"Inject Ephemeral Secret"| Pod
  Pod --> Wiz

Key Architectural Decisions & Engineering Tradeoffs

DECISION 01

Adopt External Secrets Operator (ESO) with HashiCorp Vault instead of committing encrypted secrets in Git

ARCHITECTURAL RATIONALE:

Allows application repos and GitOps manifests to contain only declarative ExternalSecret metadata pointers while actual secret payloads are fetched dynamically at runtime via Kubernetes ServiceAccount JWT authentication.

ENGINEERING TRADEOFF:

Requires high-availability Vault cluster operations and scoped SecretStore RBAC per tenant namespace.

DECISION 02

Enforce cryptographic image verification at admission time with Cosign and OPA Policy Agent

ARCHITECTURAL RATIONALE:

Prevents unauthorized or tampered images from running in production even if an engineer has deployment permissions.

ENGINEERING TRADEOFF:

All internal base images and third-party mirroring pipelines must pass through the signing workflow.

Zero-Trust Security Guardrails

  • Zero static credentials stored in Git repositories or CI logs; all runtime secrets rotate centrally in HashiCorp Vault.
  • OPA admission controllers reject unsigned container images, `latest` tags, and privileged security contexts.
  • Wiz CNAPP integration continuously maps attack paths across container images, Kubernetes RBAC permissions, and network exposure.

SRE & Reliability Engineering

  • ESO caches synced Kubernetes Secrets with configurable refresh intervals so transient Vault maintenance never disrupts running pods.
  • Harbor registry deployed on Kubernetes with HA persistence and automated replication.

Engineering Problem

Modern platform security requires defending both the delivery supply chain (what code and container images are allowed into the cluster) and the runtime control plane (how workloads authenticate to downstream databases and APIs without leaking credentials in Git).

Architectural Implementation

1. Dynamic Secrets Delivery with HashiCorp Vault & ESO

To reconcile GitOps (where everything lives in Git) with Zero Trust (where no secrets live in Git), we implemented HashiCorp Vault paired with the External Secrets Operator (ESO):

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payment-service-db-credentials
  namespace: prod-payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-enterprise-backend
    kind: ClusterSecretStore
  target:
    name: payment-db-secret
    creationPolicy: Owner
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: kv/prod/payments/database
        property: password

Workloads authenticate to Vault using their short-lived Kubernetes ServiceAccount token, and ESO reconciles the runtime secret with automatic rotation.

2. Software Supply Chain Integrity (Harbor + Trivy/Snyk + Cosign + OPA)

  1. Vulnerability Scanning: Every container image built in CI is scanned for OS and library CVEs using Snyk and Trivy and stored in a self-hosted Harbor OCI registry.
  2. Cryptographic Signing: Validated images are signed using Sigstore Cosign.
  3. Admission Gatekeeping: OPA Policy Agent verifies image signatures, prohibits mutable tags (:latest), and enforces resource requests/limits before the Kubernetes API server admits any pod.

Complete Technology Stack

HashiCorp Vault External Secrets Operator (ESO) Sigstore Cosign OPA Policy Agent Wiz CNAPP Harbor Registry Trivy Snyk OpenShift RBAC & SCC