top of page
Search

Deploying Hashicorp Vault to EKS cluster with DynamoDB backend.

Problem statement

Kubernetes secret is not the best place to keep sensitive data. Even though EKS supports adding KMS envelope encryption to enhance security for secrets, we still can not apply fine-grained control access to kubernetes secrets via RBAC. A better idea is to use an external secrets store, for example AWS Secrets Manager, Hashicorp Vault, etc. In this post we will look at deployment of Hashicorp Vault in EKS cluster in a highly-available manner with DynamoDB as backend and KMS for encryption.

Solution overview

  1. Hashicorp Vault will be deployed by Helm Chart into EKS cluster.

  2. Vault pods will assume IAM roles via IAM OIDC identity provider and IRSA.

  3. DynamoDB will be used as the backend for Vault.

  4. AWS KMS will be used for Vault auto-unseal.

  5. DynamoDB and KMS are accessible through VPC endpoints.

  6. Vault UI will be exposed via Elastic Load Balancer.

  7. Vault injector pod will set secret into application pod.

  8. Vault writes audit logs (without any sensitive data) into persistent EBS volume.

  9. Data encrypted at rest and in transit.

Hashicorp Vault supports a variety of backends including Amazon S3 and DynamoDB. Highly-available setup is possible only with DynamoDB, so we will use it.


Before installing the Vault Helm chart we need to create a kubernetes namespace:

$ kubectl create namespace vault
namespace/vault created

We can use the official helm repository:

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp"has been added to your repositories

$ helm search repo hashicorp/vault
NAME             CHART   VERSIONAPP VERSIONDESCRIPTION                   
hashicorp/vault  0.19.0  1.9.2      Official HashiCorp Vault Chart

We need to generate SSL certificates in order to configure encryption is transit:

$ NAMESPACE=vault

$ SECRET_NAME=vault-server-tls
$ SECRET_NAME=vault-server-tls
$ TMPDIR=/tmp

$ openssl genrsa -out ${TMPDIR}/vault.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
............+++++
..............+++++
e is 65537 (0x010001)


$ cat <<EOF >${TMPDIR}/csr.conf
> [req]
> req_extensions = v3_req
> distinguished_name = req_distinguished_name
> [req_distinguished_name]
> [ v3_req ]
> basicConstraints = CA:FALSE
> keyUsage = nonRepudiation, digitalSignature, keyEncipherment
> extendedKeyUsage = serverAuth
> subjectAltName = @alt_names
> [alt_names