Overview of third-party addons for EKS (Teleport)

Table of Contents

In the previous post we checked several EKS addons (Kubecost, Dynarace and Istio), but we still have others. In this post we will look at Teleport. Teleport is an identity-aware, multi-protocol access proxy which understands SSH, HTTPS, RDP, Kubernetes API, MySQL, MongoDB and PostgreSQL wire protocols.

Add-on installation

Teleport can be installed into your EKS clusted as add-on.

Once the add-on is installed, you need to find a load balancer, that was created:

$ kubectl get services -n teleport
NAME       TYPE           CLUSTER-IP      EXTERNAL-IP                                                               PORT(S)                                                                     AGE
teleport   LoadBalancer   ac211ba11cc6a4340a2012236e7fc418-1390710022.us-east-1.elb.amazonaws.com   443:31946/TCP,3023:32150/TCP,3026:32689/TCP,3024:31432/TCP,3036:31808/TCP   44s

Then you need to add the load balancer endpoint to the ConfigMap “teleport”:

$ kubectl edit cm -n teleport teleport

apiVersion: v1
  teleport.yaml: |
        severity: INFO
        output: stderr
          output: text
          extra_fields: ["timestamp","level","component","caller"]
      enabled: true
      cluster_name: teleport.teleport.svc.cluster.local
        type: "local"
        local_auth: true
        second_factor: "otp"
      public_addr: ['ac211ba11cc6a4340a2012236e7fc418-1390710022.us-east-1.elb.amazonaws.com', 'teleport.teleport.svc.cluster.local:443', '']
      enabled: true
      enabled: false
kind: ConfigMap
  creationTimestamp: "2023-02-27T12:10:47Z"
  name: teleport
  namespace: teleport
  resourceVersion: "17380"
  uid: 2da3f3b3-297f-45a8-b8b7-8d155d23ba76

Recreate all Teleport pods:

$ kubectl delete pods -n teleport --all
pod "teleport-b99977958-6hwdw" deleted

$ k get po -n teleport -w
NAME                       READY   STATUS    RESTARTS   AGE
teleport-b99977958-tkrxk   1/1     Running   0          7s

Create a new “admin” user. You will recieve a link for setting up the password and MFA (required).

$ kubectl exec -n teleport deployment/teleport -- tctl users add admin --roles=editor,access,auditor

User "admin" has been created but requires a password. Share this URL with the user to complete user setup, link is valid for 1h:

Server Access

Teleport Server Access consolidates SSH access across all environments, decreases configuration complexity, supports industry best practices and compliance while giving complete visibility over all sessions and events.

Teleport Server Access is designed for the following kinds of scenarios:

  • When up to a vast number of clusters must be managed using the command-line (tsh) or programmatically (through the Teleport API) and you want to simplify your stack, security, and configuration complexity.

  • When security team members must track and audit every user session.

  • When Teleport users require a complete, dedicated, and secure SSH option (Teleport Node running in SSH mode) and more than a certificate authority (Teleport Auth) with proxy (Teleport Proxy).

  • When resource and network security must be maximized: SSH certificates over secret keys, Two-Factor Authentication (2FA), Single Sign-On (SSO), and short-lived certificates.

Here is an example of manual adding a new VM. I chose Amazon Linux 2 EC2 instance, but Teleport supports various OSs:

A bootstrap script is generated with a short-term access token:

Script installs required packages and configure them:

sudo bash -c "$(curl -kfsSL https://ac211ba11cc6a4340a2012236e7fc418-1390710022.us-east-1.elb.amazonaws.com/scripts/3bd4fa2bd8b232302aa465f51ddee489/install-node.sh)"

2023-02-27 13:09:57 UTC [teleport-installer] TELEPORT_VERSION: 10.3.1
2023-02-27 13:09:57 UTC [teleport-installer] TARGET_HOSTNAME: ac211ba11cc6a4340a2012236e7fc418-1390710022.us-east-1.elb.amazonaws.com
2023-02-27 13:09:57 UTC [teleport-installer] TARGET_PORT: 3080
2023-02-27 13:09:57 UTC [teleport-installer] JOIN_TOKEN: 3bd4fa2bd8b232302aa465f51ddee489
2023-02-27 13:09:57 UTC [teleport-installer] CA_PIN_HASHES: sha256:47525f0632e6382036180cbeb9bd975383d3ca58fbb8f7f3f69023d2996c64d7
2023-02-27 13:09:57 UTC [teleport-installer] Checking TCP connectivity to Teleport server (ac211ba11cc6a4340a2012236e7fc418-1390710022.us-east-1.elb.amazonaws.com:3080)
2023-02-27 13:09:57 UTC [teleport-installer] Couldn't find nc, telnet or /dev/tcp to do a connection test
2023-02-27 13:09:57 UTC [teleport-installer] Going to blindly continue without testing connectivity
2023-02-27 13:09:57 UTC [teleport-installer] Detected host: linux-gnu, using Teleport binary type linux
2023-02-27 13:09:57 UTC [teleport-installer] Detected arch: x86_64, using Teleport arch amd64
2023-02-27 13:09:57 UTC [teleport-installer] Detected distro type: "centos rhel fedora"
2023-02-27 13:09:57 UTC [teleport-installer] Using Teleport distribution: rpm
2023-02-27 13:09:57 UTC [teleport-installer] Created temp dir /tmp/teleport-1ITr6q3fQS
2023-02-27 13:09:57 UTC [teleport-installer] Found 'yum' package manager, using it
2023-02-27 13:09:57 UTC [teleport-installer] Downloading Teleport rpm release 10.3.1
2023-02-27 13:09:57 UTC [teleport-installer] Running curl -fsSL --retry 5 --retry-delay 5 https://get.gravitational.com/teleport-10.3.1-1.x86_64.rpm
2023-02-27 13:09:57 UTC [teleport-installer] Downloading to /tmp/teleport-1ITr6q3fQS/teleport-10.3.1-1.x86_64.rpm
2023-02-27 13:10:07 UTC [teleport-installer] Downloaded file size: 112045092 bytes
2023-02-27 13:10:07 UTC [teleport-installer] Will use sha256sum to validate the checksum of the downloaded file
2023-02-27 13:10:08 UTC [teleport-installer] The downloaded file's checksum validated correctly
2023-02-27 13:10:08 UTC [teleport-installer] Installing Teleport release from /tmp/teleport-1ITr6q3fQS/teleport-10.3.1-1.x86_64.rpm using yum -y localinstall
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Examining /tmp/teleport-1ITr6q3fQS/teleport-10.3.1-1.x86_64.rpm: teleport-10.3.1-1.x86_64
Marking /tmp/teleport-1ITr6q3fQS/teleport-10.3.1-1.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package teleport.x86_64 0:10.3.1-1 will be installed
--> Finished Dependency Resolution
amzn2-core/2/x86_64                                                                                                                                                                        | 3.7 kB  00:00:00

Dependencies Resolved

 Package                                        Arch                                         Version                                        Repository                                                       Size
 teleport                                       x86_64                                       10.3.1-1                                       /teleport-10.3.1-1.x86_64                                       370 M

Transaction Summary
Install  1 Package

Total size: 370 M
Installed size: 370 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : teleport-10.3.1-1.x86_64                                                                                                                                                                       1/1
  Verifying  : teleport-10.3.1-1.x86_64                                                                                                                                                                       1/1

  teleport.x86_64 0:10.3.1-1

2023-02-27 13:10:15 UTC [teleport-installer] Found: Teleport v10.3.1 git:v10.3.1-0-g2fa9454 go1.18.6
2023-02-27 13:10:15 UTC [teleport-installer] Writing Teleport node service config to /etc/teleport.yaml
Wrote config to file "/etc/teleport.yaml". Now you can start the server. Happy Teleporting!
2023-02-27 13:10:15 UTC [teleport-installer] Host is using systemd
2023-02-27 13:10:15 UTC [teleport-installer] Starting Teleport via systemd. It will automatically be started whenever the system reboots.
Created symlink from /etc/systemd/system/multi-user.target.wants/teleport.service to /usr/lib/systemd/system/teleport.service.

Teleport has been started.

View its status with 'sudo systemctl status teleport.service'
View Teleport logs using 'sudo journalctl -u teleport.service'
To stop Teleport, run 'sudo systemctl stop teleport.service'
To start Teleport again if you stop it, run 'sudo systemctl start teleport.service'

You can see this node connected in the Teleport web UI or 'tsh ls' with the name 'ip-10-0-151-153.ec2.internal'
Find more details on how to use Teleport here: https://goteleport.com/docs/user-manual/

Choose the OS user that is present in the VM and will be used by Teleport:

And test a connectivity:

After that you can see the VM in the list:

You don’t need to register all EC2 instances manually, there is a native capability to do it automatically.

The Teleport Discovery Service can connect to Amazon EC2 and automatically discover and enroll EC2 instances matching configured labels. It will then execute an install script on these discovered instances using AWS Systems Manager that will install Teleport, start it and join the cluster.

You can get a CLI via your web browser:

or your local terminal:

$ tctl version
Teleport v12.0.5 git:api/v12.0.5-0-g818318c go1.20.1

$ tsh version
Teleport v12.0.5 git:api/v12.0.5-0-g818318c go1.20.1

$ tsh login --proxy=teleport.ait-demo1.com --user=admin
Enter password for Teleport user admin:
Enter an OTP code from a device:

> Profile URL:        https://teleport.ait-demo1.com:443
  Logged in as:       admin
  Cluster:            teleport.ait-demo1.com
  Roles:              access, editor
  Logins:             ec2-user, -teleport-internal-join
  Kubernetes:         enabled
  Valid until:        2023-03-05 09:23:03 +0200 EET [valid for 12h0m0s]
  Extensions:         login-ip, permit-agent-forwarding, permit-port-forwarding, permit-pty, private-key-policy

$ tsh ls
Node Name                    Address    Labels                                                                                        
ip-10-0-151-153.ec2.internal <- Tunnel   hostname=ip-10-0-151-153.ec2.internal,teleport.internal/resource-id=39162ea2-7e84-4be1-a417-d416f2d3b... 

$ tsh ssh ec2-user@ip-10-0-151-153.ec2.internal

[ec2-user@ip-10-0-151-153 ~]$ ls -la
total 12
drwx------ 3 ec2-user ec2-user  74 Feb 27 12:23 .
drwxr-xr-x 4 root     root      38 Feb 27 13:08 ..
-rw-r--r-- 1 ec2-user ec2-user  18 Jul 15  2020 .bash_logout
-rw-r--r-- 1 ec2-user ec2-user 193 Jul 15  2020 .bash_profile
-rw-r--r-- 1 ec2-user ec2-user 231 Jul 15  2020 .bashrc
drwx------ 2 ec2-user ec2-user  29 Feb 27 12:23 .ssh

Other users can join the active session as “Observer”, “Moderator” or “Peer”

  • peer: Can join and collaborate in a session. They can view output and send input.

  • moderator: Can join and watch a session. They can view output and forcefully terminate the session at will.

  • observer: Can join and watch a session. They cannot control the session in any way.

    Access logs usually contain information like:

    • IP address or hostname of the client

    • Timestamps of operations or access attempts

    • Event metadata like “severity”

    • Result of an attempted operation (success/failure)

    • Log messaging output from the operation

    It’s incredibly important to understand what types of data are being radiated and recorded by your system so that you can stay on top of your infrastructure.

    Kubernetes access

    Teleport provides secure access to Kubernetes clusters:

    • Users can access Kubernetes clusters with Single Sign-On (SSO) providers like Okta and switch between clusters without logging in twice.

    • Operators can implement granular role-based access controls, including limiting access to specific Kubernetes clusters or even specific pods within a namespace.

    • Organizations can achieve compliance by recording kubectl sessions.

    I chose EKS cluster for demo. You will need to install the Teleport agent Helm chart, values file will be generated for you:

    Values file contains a shotr-term auth token:

    $ helm install teleport-agent teleport/teleport-kube-agent -f prod-cluster-values.yaml --create-namespace --namespace teleport
    $ kubectl get po teleport-agent-0 -n teleport
    NAME               READY   STATUS    RESTARTS   AGE
    teleport-agent-0   1/1     Running   0          28s

    When your Kubernetes cluster is identified, you can connect:

    $ tsh kube login demo1
    Logged into Kubernetes cluster "demo1". Try 'kubectl version' to test the connection.
    $ kubectl get pods --all-namespaces
    NAMESPACE     NAME                                  READY   STATUS    RESTARTS        AGE
    development   loadbalancer-6df4b7468c-wzbmx         1/1     Running   0               17m
    development   webapp-66b94c9c64-ms4r6               1/1     Running   0               17m
    kube-system   aws-node-87f82                        1/1     Running   0               5d9h
    kube-system   aws-node-vq5ft                        1/1     Running   0               5d9h
    kube-system   coredns-79989457d9-jw88j              1/1     Running   0               5d9h
    kube-system   coredns-79989457d9-p5p6j              1/1     Running   0               5d9h
    kube-system   ebs-csi-controller-68fb47d4f5-4gmqt   6/6     Running   1 (5h42m ago)   5d9h
    kube-system   ebs-csi-controller-68fb47d4f5-qtm2l   6/6     Running   0               5d9h
    kube-system   ebs-csi-node-jkjd2                    3/3     Running   0               5d9h
    kube-system   ebs-csi-node-zclxv                    3/3     Running   0               5d9h
    kube-system   kube-proxy-5lsc6                      1/1     Running   0               5d9h
    kube-system   kube-proxy-gb5fb                      1/1     Running   0               5d9h
    production    loadbalancer-6df4b7468c-7khv6         1/1     Running   0               17m
    production    webapp-66b94c9c64-p5n5c               1/1     Running   0               17m

    EKS clusters can be registered in Teleport automatially as well
    EKS Auto-Discovery can automatically discover any EKS cluster and enroll it in Teleport if its tags match the configured labels.

    Teleport Kubernetes Auto-Discovery involves two components.

    The first, the Discovery Service, is responsible for watching your cloud provider and checking if there are any new clusters or if there have been any modifications to previously discovered clusters. The second, the Kubernetes Service, monitors the clusters created by the Discovery Service. It proxies communications between users and the API servers of these clusters.

    Database Access

    Teleport can provide secure connections to your databases while improving both access control and visibility.

    Some of the things you can do with Database Access:

    • Enable users to retrieve short-lived database certificates using a Single Sign-On flow, thus maintaining their organization-wide identity.

    • Configure role-based access controls for databases and implement custom Access Request workflows.

    • Capture database activity in the Teleport audit log.

    Teleport supports many types of databases:

    • Active Directory SQL Server: Connect Microsoft SQL Server with Active Directory authentication.

    • AWS DynamoDB: Connect AWS DynamoDB.

    • AWS ElastiCache & MemoryDB: Connect AWS ElastiCache or AWS MemoryDB for Redis database.

    • AWS RDS & Aurora: Connect AWS RDS or Aurora PostgreSQL, MariaDB or MySQL database.

    • AWS RDS Proxy: Connect AWS RDS Proxy instances to Teleport.

    • AWS Redshift: Connect AWS Redshift database.

    • AWS Redshift Serverless: Connect to AWS Redshift serverless.

    • AWS Keyspaces (Apache Cassandra): Connect to an AWS Keyspaces database.

    • Azure PostgreSQL & MySQL: Connect Azure PostgreSQL or MySQL.

    • Azure Cache for Redis: Connect Azure Cache for Redis.

    • Azure SQL Server: Connect Azure SQL Server with Azure Active Directory authentication.

    • GCP Cloud SQL MySQL: Connect GCP Cloud SQL MySQL database.

    • GCP Cloud SQL PostgreSQL: Connect GCP Cloud SQL PostgreSQL database.

    • MongoDB Atlas: Connect MongoDB Atlas cluster.

    • Self-hosted CockroachDB: Connect self-hosted CockroachDB database.

    • Self-hosted Elasticsearch

    • Self-hosted MongoDB: Connect self-hosted MongoDB database.

    • Self-hosted MySQL & MariaDB: Connect self-hosted MySQL or MariaDB database.

    • Self-hosted PostgreSQL: Connect self-hosted PostgreSQL database.

    • Self-hosted Redis Cluster: Connect a self-hosted Redis Cluster.

    • Self-hosted Redis: Connect self-hosted Redis.

    • Self-Hosted Cassandra & ScyllaDB: Connect self-hosted Cassandra or ScyllaDB.

    • Snowflake: Connect Snowflake.

    For this demo I chose AWS RDS for MySQL. A high-level diagram if the following:

    Add the database as all other services:

    Choose the required type:

    Enter the DB connection endpoint, AWS account ID and RDS Resource ID:

    The generated bootstrap script should be executed on the host that has a network connectivity wiyh your RDS:

    RDS IAM Authentication will be used for login, so the intermediate host must have such permissions:

    DB user and database should be created in advance. AWSAuthenticationPlugin should be enabled and appropriate permissions granted

    CREATE USER demouser IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';
    GRANT ALL ON `%`.* TO 'demouser'@'%';

    Then you can use your local terminal to connect to the database, even if it is deployed in a private subnet.

    $ tsh db connect demodb --db-user=demouser --db-name=demodb
    mysql> show databases;
    | Database           |
    | demodb             |
    | information_schema |
    | mysql              |
    | performance_schema |
    | sys                |
    5 rows in set (0.13 sec)


    There are two pricing models, Community and Enterprise:

    GitHub link


    Teleport is the identity-native infrastructure access platform for engineers and machines. By replacing insecure secrets like passwords, keys and tokens with true identity based on biometrics and security modules, Teleport delivers phishing-proof zero trust for every engineer and service connected to your global infrastructure. The open-source Teleport Access Platform consolidates connectivity, authentication, authorization and audit into a single source of truth for access policy across your entire infrastructure while delivering a frictionless developer experience. Teleport replaces VPNs, shared credentials, secrets vaults and legacy PAM solutions, improving security and engineering productivity.

    Available in the AWS Marketplace as EKS add-on