Deploying Hashicorp Vault to EKS cluster with DynamoDB backend.

Updated: Aug 30

Problem statement

Kubernetes secret is not the best place to keep sensitive data. Even though EKS supports adding KMS envelope encryption to enhance security for secrets, we still can not apply fine-grained control access to kubernetes secrets via RBAC. A better idea is to use an external secrets store, for example AWS Secrets Manager, Hashicorp Vault, etc. In this post we will look at deployment of Hashicorp Vault in EKS cluster in a highly-available manner with DynamoDB as backend and KMS for encryption.

Solution overview

  1. Hashicorp Vault will be deployed by Helm Chart into EKS cluster.

  2. Vault pods will assume IAM roles via IAM OIDC identity provider and IRSA.

  3. DynamoDB will be used as the backend for Vault.

  4. AWS KMS will be used for Vault auto-unseal.

  5. DynamoDB and KMS are accessible through VPC endpoints.

  6. Vault UI will be exposed via Elastic Load Balancer.

  7. Vault injector pod will set secret into application pod.

  8. Vault writes audit logs (without any sensitive data) into persistent EBS volume.

  9. Data encrypted at rest and in transit.

Hashicorp Vault supports a variety of backends including Amazon S3 and DynamoDB. Highly-available setup is possible only with DynamoDB, so we will use it.


Before installing the Vault Helm chart we need to create a kubernetes namespace:

$ kubectl create namespace vault
namespace/vault created

We can use the official helm repository:

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp"has been added to your repositories

$ helm search repo hashicorp/vault
NAME             CHART   VERSIONAPP VERSIONDESCRIPTION                   
hashicorp/vault  0.19.0  1.9.2      Official HashiCorp Vault Chart

We need to generate SSL certificates in order to configure encryption is transit:

$ NAMESPACE=vault

$ SECRET_NAME=vault-server-tls
$ SECRET_NAME=vault-server-tls
$ TMPDIR=/tmp

$ openssl genrsa -out ${TMPDIR}/vault.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
............+++++
..............+++++
e is 65537 (0x010001)


$ cat <<EOF >${TMPDIR}/csr.conf
> [req]
> req_extensions = v3_req
> distinguished_name = req_distinguished_name
> [req_distinguished_name]
> [ v3_req ]
> basicConstraints = CA:FALSE
> keyUsage = nonRepudiation, digitalSignature, keyEncipherment
> extendedKeyUsage = serverAuth
> subjectAltName = @alt_names
> [alt_names]
> DNS.1 = ${SERVICE}
> DNS.2 = ${SERVICE}.${NAMESPACE}
> DNS.3 = ${SERVICE}.${NAMESPACE}.svc
> DNS.4 = ${SERVICE}.${NAMESPACE}.svc.cluster.local
> IP.1 = 127.0.0.1
> EOF


$ openssl req -new -key ${TMPDIR}/vault.key -subj "/CN=${SERVICE}.${NAMESPACE}.svc" -out ${TMPDIR}/server.csr -config ${TMPDIR}/csr.conf

Next we need to create a certificate signing request in kubernetes and approve it:

$ export CSR_NAME=vault-csr

$ cat <<EOF >${TMPDIR}/csr.yaml
> apiVersion: certificates.k8s.io/v1beta1
> kind: CertificateSigningRequest
> metadata:
>   name: ${CSR_NAME}
> spec:
>   groups:
>   - system:authenticated
>   request: $(cat ${TMPDIR}/server.csr | base64 | tr -d '\n')
>   usages:
>   - digital signature
>   - key encipherment
>   - server auth
> EOF


$ kubectl create -f ${TMPDIR}/csr.yaml

$ kubectl get csr
NAME        AGE   SIGNERNAME                     REQUESTOR          CONDITION
vault-csr   9s    kubernetes.io/legacy-unknown   kubernetes-admin   Pending


$ kubectl certificate approve ${CSR_NAME}
certificatesigningrequest.certificates.k8s.io/vault-csr approved


$ kubectl get csr
NAME        AGE   SIGNERNAME                     REQUESTOR          CONDITION
vault-csr   18s   kubernetes.io/legacy-unknown   kubernetes-admin   Approved,Issued

Next we need to create a kubernetes secret with certificate and key:

$ serverCert=$(kubectl get csr ${CSR_NAME} -o jsonpath='{.status.certificate}')


$ echo "${serverCert}" | openssl base64 -d -A -out ${TMPDIR}/vault.crt

$ kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 -d > ${TMPDIR}/vault.ca


$ kubectl create secret generic ${SECRET_NAME} \
>         --namespace ${NAMESPACE} \
>         --from-file=vault.key=${TMPDIR}/vault.key \
>         --from-file=vault.crt=${TMPDIR}/vault.crt \
>         --from-file=vault.ca=${TMPDIR}/vault.ca
secret/vault-server-tls created

We need to override some chart values:

override-values.yml

# Vault Helm Chart Value Overrides
global:
  enabled: true
  tlsDisable: false

injector:
  enabled: true
# Use the Vault K8s Image https://github.com/hashicorp/vault-k8s/
  image:
    repository:"hashicorp/vault-k8s"
    tag:"0.14.2"

server:
# Use the Enterprise Image
  image:
    repository:"hashicorp/vault"
    tag:"1.9.2"

  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca

  extraVolumes:
    - type: secret
      name: vault-server-tls

# This configures the Vault Statefulset to create a PVC for audit logs.
# See https://www.vaultproject.io/docs/audit/index.html to know more
  auditStorage:
    enabled: true

  standalone:
    enabled: false

  serviceAccount:
    create: false
    name:"vault-sa"

# Run Vault in "HA" mode.
  ha:
    enabled: true
    replicas:5

    config: |
      ui = true
      listener "tcp" {
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
        tls_key_file = "/vault/userconfig/vault-server-tls/vault.key"
        tls_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
      }

# S3 example. Not used in the solution
# storage "s3" {
#   bucket     = "vault-ait-bucket"
#   region     = "us-east-1"
#   kms_key_id = "alias/aws/s3"
# }

      storage "dynamodb" {
        ha_enabled = "true"
        region = "us-east-1"
        table = "vault-table"
      }

      seal "awskms" {
        region     = "us-east-1"
        kms_key_id = "ceedc5f5-7963-4482-a9df-17f34fad8169"
endpoint   = "https://vpce-06a1058e44fb409e5-3e7zjquu.kms.us-east-1.vpce.amazonaws.com"
      }

      service_registration "kubernetes" {}

# Vault UI
ui:
  enabled: true
  serviceType:"LoadBalancer"
  serviceNodePort: null
  externalPort:8200

# For Added Security, edit the below
#loadBalancerSourceRanges:
#   - < Your IP RANGE Ex. 10.0.0.0/16 >
#   - < YOUR SINGLE IP Ex. 1.78.23.3/32 >

Vault pods will use serviceAccount “vault-sa” and assume appropriate IAM role according to the least privilege principle.