Problem statement
Kubernetes secret is not the best place to keep sensitive data. Even though EKS supports adding KMS envelope encryption to enhance security for secrets, we still can not apply fine-grained control access to kubernetes secrets via RBAC. A better idea is to use an external secrets store, for example AWS Secrets Manager, Hashicorp Vault, etc. In this post we will look at deployment of Hashicorp Vault in EKS cluster in a highly-available manner with DynamoDB as backend and KMS for encryption.
Solution overview
- Hashicorp Vault will be deployed by Helm Chart into EKS cluster.
- Vault pods will assume IAM roles via IAM OIDC identity provider and IRSA.
- DynamoDB will be used as the backend for Vault.
- AWS KMS will be used for Vault auto-unseal.
- DynamoDB and KMS are accessible through VPC endpoints.
- Vault UI will be exposed via Elastic Load Balancer.
- Vault injector pod will set secret into application pod.
- Vault writes audit logs (without any sensitive data) into persistent EBS volume.
- Data encrypted at rest and in transit.
Hashicorp Vault supports a variety of backends including Amazon S3 and DynamoDB. Highly-available setup is possible only with DynamoDB, so we will use it.
Before installing the Vault Helm chart we need to create a kubernetes namespace:
$ kubectl create namespace vault namespace/vault created
We can use the official helm repository:
$ helm repo add hashicorp https://helm.releases.hashicorp.com "hashicorp"has been added to your repositories $ helm search repo hashicorp/vault NAME CHART VERSIONAPP VERSIONDESCRIPTION hashicorp/vault 0.19.0 1.9.2 Official HashiCorp Vault Chart
We need to generate SSL certificates in order to configure encryption is transit:
$ NAMESPACE=vault
$ SECRET_NAME=vault-server-tls
$ SECRET_NAME=vault-server-tls
$ TMPDIR=/tmp
$ openssl genrsa -out ${TMPDIR}/vault.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
............+++++
..............+++++
e is 65537 (0x010001)
$ cat <<EOF >${TMPDIR}/csr.conf
> [req]
> req_extensions = v3_req
> distinguished_name = req_distinguished_name
> [req_distinguished_name]
> [ v3_req ]
> basicConstraints = CA:FALSE
> keyUsage = nonRepudiation, digitalSignature, keyEncipherment
> extendedKeyUsage = serverAuth
> subjectAltName = @alt_names
> [alt_names]
> DNS.1 = ${SERVICE}
> DNS.2 = ${SERVICE}.${NAMESPACE}
> DNS.3 = ${SERVICE}.${NAMESPACE}.svc
> DNS.4 = ${SERVICE}.${NAMESPACE}.svc.cluster.local
> IP.1 = 127.0.0.1
> EOF
$ openssl req -new -key ${TMPDIR}/vault.key -subj "/CN=${SERVICE}.${NAMESPACE}.svc" -out ${TMPDIR}/server.csr -config ${TMPDIR}/csr.conf
Next we need to create a certificate signing request in kubernetes and approve it:
$ export CSR_NAME=vault-csr
$ cat <<EOF >${TMPDIR}/csr.yaml
> apiVersion: certificates.k8s.io/v1beta1
> kind: CertificateSigningRequest
> metadata:
> name: ${CSR_NAME}
> spec:
> groups:
> - system:authenticated
> request: $(cat ${TMPDIR}/server.csr | base64 | tr -d '\n')
> usages:
> - digital signature
> - key encipherment
> - server auth
> EOF
$ kubectl create -f ${TMPDIR}/csr.yaml
$ kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
vault-csr 9s kubernetes.io/legacy-unknown kubernetes-admin Pending
$ kubectl certificate approve ${CSR_NAME}
certificatesigningrequest.certificates.k8s.io/vault-csr approved
$ kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
vault-csr 18s kubernetes.io/legacy-unknown kubernetes-admin Approved,Issued
Next we need to create a kubernetes secret with certificate and key:
$ serverCert=$(kubectl get csr ${CSR_NAME} -o jsonpath='{.status.certificate}')
$ echo "${serverCert}" | openssl base64 -d -A -out ${TMPDIR}/vault.crt
$ kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 -d > ${TMPDIR}/vault.ca
$ kubectl create secret generic ${SECRET_NAME} \
> --namespace ${NAMESPACE} \
> --from-file=vault.key=${TMPDIR}/vault.key \
> --from-file=vault.crt=${TMPDIR}/vault.crt \
> --from-file=vault.ca=${TMPDIR}/vault.ca
secret/vault-server-tls created
We need to override some chart values:
override-values.yml
# Vault Helm Chart Value Overrides
global:
enabled: true
tlsDisable: false
injector:
enabled: true
# Use the Vault K8s Image https://github.com/hashicorp/vault-k8s/
image:
repository:"hashicorp/vault-k8s"
tag:"0.14.2"
server:
# Use the Enterprise Image
image:
repository:"hashicorp/vault"
tag:"1.9.2"
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca
extraVolumes:
- type: secret
name: vault-server-tls
# This configures the Vault Statefulset to create a PVC for audit logs.
# See https://www.vaultproject.io/docs/audit/index.html to know more
auditStorage:
enabled: true
standalone:
enabled: false
serviceAccount:
create: false
name:"vault-sa"
# Run Vault in "HA" mode.
ha:
enabled: true
replicas:5
config: |
ui = true
listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
tls_key_file = "/vault/userconfig/vault-server-tls/vault.key"
tls_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
}
# S3 example. Not used in the solution
# storage "s3" {
# bucket = "vault-ait-bucket"
# region = "us-east-1"
# kms_key_id = "alias/aws/s3"
# }
storage "dynamodb" {
ha_enabled = "true"
region = "us-east-1"
table = "vault-table"
}
seal "awskms" {
region = "us-east-1"
kms_key_id = "ceedc5f5-7963-4482-a9df-17f34fad8169"
endpoint = "https://vpce-06a1058e44fb409e5-3e7zjquu.kms.us-east-1.vpce.amazonaws.com"
}
service_registration "kubernetes" {}
# Vault UI
ui:
enabled: true
serviceType:"LoadBalancer"
serviceNodePort: null
externalPort:8200
# For Added Security, edit the below
#loadBalancerSourceRanges:
# - < Your IP RANGE Ex. 10.0.0.0/16 >
# - < YOUR SINGLE IP Ex. 1.78.23.3/32 >
Vault pods will use serviceAccount “vault-sa” and assume appropriate IAM role according to the least privilege principle.
IAM role trust policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::3**********3:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/0F51B5F333F6947BC27C138A1BC3E135"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/0F51B5F333F6947BC27C138A1BC3E135:sub": "system:serviceaccount:vault:vault-sa"
}
}
}
]
}
Policies have to allow KMS and DynamoDB access:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowKMS",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:Encrypt",
"kms:DescribeKey"
],
"Resource": "arn:aws:kms:us-east-1:3**********3:key/ceedc5f5-7963-4482-a9df-17f34fad8169"
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"dynamodb:DescribeLimits",
"dynamodb:DescribeTimeToLive",
"dynamodb:ListTagsOfResource",
"dynamodb:DescribeReservedCapacityOfferings",
"dynamodb:DescribeReservedCapacity",
"dynamodb:ListTables",
"dynamodb:BatchGetItem",
"dynamodb:BatchWriteItem",
"dynamodb:CreateTable",
"dynamodb:DeleteItem",
"dynamodb:GetItem",
"dynamodb:GetRecords",
"dynamodb:PutItem",
"dynamodb:Query",
"dynamodb:UpdateItem",
"dynamodb:Scan",
"dynamodb:DescribeTable"
],
"Effect": "Allow",
"Resource": [
"arn:aws:dynamodb:us-east-1:3**********3:table/vault-table"
]
}
]
}
Kubernetes service account has the following annotation:
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::3**********3:role/vault-role
name: vault-sa
namespace: vault
Next we can install the helm chart
$ helm install vault hashicorp/vault --namespace vault -f override-values.yml $ kubectl get pods -n vault vault vault-0 0/1 Running 0 14m vault vault-1 0/1 Running 0 14m vault vault-2 0/1 Running 0 14m vault vault-3 0/1 Running 0 14m vault vault-4 0/1 Running 0 14m vault vault-agent-injector-f4c4f579c-b2lv8 1/1 Running 0 14m
Here we need to initialize the vault. It’s required just once, because we have auto-unseal with AWS KMS. If a Vault pod is restarted, unseal will happen automatically next time.
This feature enables operators to delegate the unsealing process to trusted cloud providers to ease operations in the event of partial failure and to aid in the creation of new or ephemeral clusters.
$ kubectl exec -n vault -it vault-0 -- /bin/sh / $ vault status Key Value --- ----- Recovery Seal Type awskms Initialized false Sealed true Total Recovery Shares 0 Threshold 0 Unseal Progress 0/0 Unseal Nonce n/a Version 1.9.2 Storage Type dynamodb HA Enabled true / $ vault operator init Recovery Key 1: WrM/CbSByO7L4wab0IK/TgFJEPMWQSz5VMHaVu6ADc/q Recovery Key 2: XJjZw4K7UX60GMfgK3kj4zOrFH0TPomiLn1BfY3wj7/1 Recovery Key 3: WI8qkKNbQvn1mMOb8QrSYaazaek6ZDYUdKVXlZJAfh8L Recovery Key 4: GvyL2vXvrHz9hPszhOaAfxSt67Va8loeVIqQpx7DtTsO Recovery Key 5: /8QFZzZkhVc9DhmtDB8rwGfewgXJxLvF+/ys8fZIZZxy Initial Root Token: s.BoYAMCWvueOavJC7zBbEhcs2 Success! Vault is initialized Recovery key initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above. / $ vault status Key Value --- ----- Recovery Seal Type shamir Initialized true Sealed false Total Recovery Shares 5 Threshold 3 Version 1.9.2 Storage Type dynamodb Cluster Name vault-cluster-b2196873 Cluster ID 245e73b0-718a-e41f-3a8e-d9d05b80f774 HA Enabled true HA Cluster https://vault-0.vault-internal:8201 HA Mode active Active Since 2022-01-25T19:21:32.839644408Z $ kubectl get pods -n vault NAME READY STATUS RESTARTS AGE vault-0 1/1 Running 0 5m vault-1 1/1 Running 0 5m vault-2 1/1 Running 0 5m vault-3 1/1 Running 0 5m vault-4 1/1 Running 0 5m vault-agent-injector-f4c4f579c-b2lv8 1/1 Running 0 8m
We have received Initial Root Token: s.BoYAMCWvueOavJC7zBbEhcs2 below, that can be used for the first login to the Vault and creating other users, groups, configurations, etc.
As an example I used web IU for creation a new secret
We can use CLI, API and Web UI for interaction with Vault. Let’s get the secret value from CLI:
/ $ vault kv get cubbyhole/prod/database/password ====== Data ====== Key Value --- ----- password Supersecret123
Everything in Vault is “path based” and we can see it in DynamoDB table:
And all sensitive data is encrypted before storing in the backend, so we can not see actual secrets value:
If the backend is S3 bucket, a file with a secret looks like:
Audit logs of a secret store are very important for the production environment. The full list of production recommendations can be found here. In the Helm chart values above we set auditStorage. In our case Persistent volume claims are created and appropriate EBS volumes are provisioned and attached to Vault pods.
$ kubectl get pvc -n vault NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE audit-vault-0 Bound pvc-0a2cc7aa-*** 10Gi RWO gp2 27h audit-vault-1 Bound pvc-355da9c0-*** 10Gi RWO gp2 27h audit-vault-2 Bound pvc-50b06581-*** 10Gi RWO gp2 27h audit-vault-3 Bound pvc-01764bd1-*** 10Gi RWO gp2 27h audit-vault-4 Bound pvc-b138651e-*** 10Gi RWO gp2 27h data-vault-0 Bound pvc-da0a285c-*** 10Gi RWO gp2 27h
/ $ df -h Filesystem Size Used Available Use% Mounted on .................................................................... /dev/nvme1n1 9.7G 36.0M 9.7G 0% /vault/audit ....................................................................
Next we need to enable the audit in the Vault once.
/ $ vault audit enable file file_path=/vault/audit/vault_audit.log Success! Enabled the file audit device at: file/ / $ vault audit list Path Type Description ---- ---- ----------- file/ file n/a
Then we can monitor activities and requests to Vault. We can see a timestamp, type of request, who did what and where.
$ kubectl exec -n vault -it vault-1 -- cat /vault/audit/vault_audit.log
....................
{"time":"2022-01-26T19:21:30.398813219Z","type":"request","auth":{"token_type":"default"},"request":{"id":"967fe6f0-8879-58f4-8dec-fb3b65279cbc","operation":"update","namespace":{"id":"root"},"path":"sys/audit/test"}}
...................
{"time":"2022-01-26T19:22:24.842012415Z","type":"request","auth":{"client_token":"hmac-sha256:712af35fc6509fc1eeda100c11d35a8aff1974c4abcd776694c2673bb3689096","accessor":"hmac-sha256:84108d0ee8c840e62ff1e7e8f2302defd9640ed6198c636da3f5820c82d2fcdd","display_name":"root","policies":["root"],"token_policies":["root"],"token_type":"service","token_issue_time":"2022-01-25T19:21:31Z"},"request":{"id":"bccb1d10-20fe-878a-481d-2535a581c7dd","operation":"read","mount_type":"system","client_token":"hmac-sha256:712af35fc6509fc1eeda100c11d35a8aff1974c4abcd776694c2673bb3689096","client_token_accessor":"hmac-sha256:84108d0ee8c840e62ff1e7e8f2302defd9640ed6198c636da3f5820c82d2fcdd","namespace":{"id":"root"},"path":"sys/internal/ui/mounts/internal/database/config","remote_address":"127.0.0.1"}}
...................
The process of secret injection is well described in official Vault documentation and no need to write it here.
Conclusion
There are many different external secrets storages that can be used for Kubernetes applications, one of them is Hashicorp Vault. Vault has the enterprise and open-source versions, and can be deployed to different clouds, platforms and operating systems. In this post we looked at deployment Vault in EKS cluster by Helm chart, with DynamoDB as backend, KMS for auto-unseal and EBS persistent volumes for audit logs. Vault was deployed in a highly available manner, data encrypted in transit and at rest, but as we saw sometimes it requires a number of manual steps. In the posts we will check other solutions, for example AWS Secrets and Configuration Provider (ASCP).
