Cross-account monitoring with Amazon CloudWatch

Problem statement

Using multiple AWS accounts is one of guidelines to set up a well-architected environment. By using multiple accounts, you can best support your security goals and business processes. In previous posts we already covered multi-account logging, backups and security services integrations. In this post we will look at multi-account monitoring with Amazon CloudWatch, because it’s important to provide a support team with ability to see metrics and dashboards from different accounts and regions in one place, for example in the dedicated “monitoring” account. Moreover we need to figure out how to configure cross-account access for dozens and hundreds of accounts and automatically add a new account for monitoring right after enrollment. Cross-account management is a best practice in ASW and Automat-IT, the post is based on knowledge gained in many implementations for our customers.

Solution design

A high-level diagram of the proposed solution is below:

Every member account will have an IAM role for cross-account access that allows the “monitoring” account to assume it. Monitoring team has access only to CloudWatch in this account. The solution has to solve the following:

• Complete configuration on 50+ current accounts
• Every new account in the organization must automatically deploy the required IAM role.
• Needs to be managed via Git repository.

Solution implementation

The recommended way for deploying resources across different AWS accounts and regions is using CloudFormation StackSets. There is a solution called “Customizations for AWS Control Tower”, which give us exactly what we need:

• Configuration can be stored as code in Git repo.
• Deployment can be started by a commit or event of AWS account enrollment.
• Resources can be deployed across different accounts and regions within a Landing Zone.

The diagram below presents the architecture of the “Customizations for AWS Control Tower”:

We need to deploy the IAM role for cross-account access for Amazon CloudWatch. The following CloudFormation template will be used:

---
AWSTemplateFormatVersion: '2010-09-09'
Description: EnablesCloudWatchincentralmonitoringaccountstoassumepermissionstoviewCloudWatchdatainthecurrentaccount

Parameters:
  MonitoringAccountIds:
    Description: Allowsoneormoremonitoringaccountstoviewyourdata.EnterAWSaccountids,12numericdigitsincomma-separatedlist
    Type: CommaDelimitedList

  Policy:
    Description: ThelevelofaccesstogivetotheMonitoringaccounts
    Type: String
    Default: CloudWatch-and-AutomaticDashboards
    AllowedValues:
      - CloudWatch-and-AutomaticDashboards
      - CloudWatch-and-ServiceLens
      - CloudWatch-AutomaticDashboards-and-ServiceLens
      - CloudWatch-core-permissions
      - View-Access-for-all-services

Conditions:
  DoFullReadOnly: !Equals [ !RefPolicy, View-Access-for-all-services ]
  DoAutomaticDashboards: !Equals [ !RefPolicy, CloudWatch-and-AutomaticDashboards ]
  DoServiceLens: !Equals [ !RefPolicy, CloudWatch-and-ServiceLens ]
  DoServiceLensAndAutomaticDashboards: !Equals [ !RefPolicy, CloudWatch-AutomaticDashboards-and-ServiceLens ]
  DoCWReadOnly: !Equals [ !RefPolicy, CloudWatch-core-permissions ]

Resources:
  CWCrossAccountSharingRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: CloudWatch-CrossAccountSharingRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              AWS: !Split
                - ','
                - !Sub
                  - 'arn:aws:iam::${inner}:root'
                  - inner: !Join
                      - ':root,arn:aws:iam::'
                      - Ref: MonitoringAccountIds
            Action:
              - sts:AssumeRole
      Path: "/"
      ManagedPolicyArns: !If
        - DoFullReadOnly
        -
          - arn:aws:iam::aws:policy/CloudWatchReadOnlyAccess
          - arn:aws:iam::aws:policy/CloudWatchAutomaticDashboardsAccess
          - arn:aws:iam::aws:policy/job-function/ViewOnlyAccess
          - arn:aws:iam::aws:policy/AWSXrayReadOnlyAccess
        - !If
          - DoAutomaticDashboards
          -
            - arn:aws:iam::aws:policy/CloudWatchReadOnlyAccess
            - arn:aws:iam::aws:policy/CloudWatchAutomaticDashboardsAccess
          - !If
            - DoServiceLens
            -
              - arn:aws:iam::aws:policy/CloudWatchReadOnlyAccess
              - arn:aws:iam::aws:policy/AWSXrayReadOnlyAccess
            - !If
              - DoServiceLensAndAutomaticDashboards
              -
                - arn:aws:iam::aws:policy/CloudWatchReadOnlyAccess
                - arn:aws:iam::aws:policy/CloudWatchAutomaticDashboardsAccess
                - arn:aws:iam::aws:policy/AWSXrayReadOnlyAccess
              -
                - arn:aws:iam::aws:policy/CloudWatchReadOnlyAccess

This template provides us exactly the role which we can see in CloudWatch settings page, if we configure cross-account monitoring manually:

The simplest configuration in the git repository look like this:

The “templates” folder consists of the above mentioned CloudFormation template. The “manifest.yaml” file looks like this:

---
region: us-east-1
version: 2021-03-15

# Control Tower Custom Resources (Service Control Policies or CloudFormation)
resources:

# A role for Cloudwatch, needed for cross-account/cross-region access to CloudWatch metrics.
  - name: MonitoringRole
    resource_file: templates/monitoring-role.yaml
    deploy_method: stack_set
    parameters:
      - parameter_key: MonitoringAccountIds
        parameter_value: "1**********4"
      - parameter_key: Policy
        parameter_value: "View-Access-for-all-services"
    deployment_targets:
       organizational_units:
         - Prod
         - Dev
         - Stage

The CodePipeline looks as following:

• The first stage is checking out the configuration code.
• The second stage is a validation of syntax and security scan, which disallow overly permissive policies, security groups, disabled encryption for databases, etc.
• The third stage deploys service control policies if you provide them.
• The last stage deploys CloudFormation templates via CloudFormation StackSets.

Once the pipeline is deployed, we can see all our AWS accounts as stack instances:

The last step in our configuration is enabling the account selector in the CloudWatch console settings of the “monitoring” account.

Now we can select different accounts and regions to see dashboards:

And you can build your own dashboards containing metrics from different accounts:

Conclusion

One of the main ideas behind using multiple AWS accounts is isolation. We provide different people with access to different accounts according to their responsibilities, for example IT department has access to “networking” account, security team has access to the “Audit” account, every developer can have his/her own sandbox account and the support team can get all needed metrics and dashboards for monitoring different environments without direct access to those accounts. In the Automat-IT Landing Zone solution we follow such best practices and encourage others to do so.