Former2 - AWS Infrastructure to CloudFormation

with a special guest: Ian McKay, Former2 Developer

While working on AWS services, I always wondered why there is no tool that I can use to

convert existing services/configurations to CloudFormation templates. It would be so practical to

have this as an option on any given AWS service console, so I asked AWS Awesome support

and our TAM several times, but unfortunately I always received the reply stating that this is a

great idea and that was the end of it.

When I finally found such a tool, my joy knew no bounds. Behold “Former2”.

I was so impressed by Ian’s solution that I contacted him by email and eventually grabbed him

for a chat about his project:

Tell me a little bit about yourself:

I am working in Sydney as the Cloud Lead at Kablamo, and also as an AWS Hero. I enjoy

speaking at meetups, co-hosting podcasts, and writing my thoughts about AWS on my blog at

onecloudplease.com.

What can you tell me about the project?

Former2 allows you to generate Infrastructure as Code outputs from your existing resources

within your AWS account (that's 124 services, mind you (-; ).

By making the relevant calls using the AWS JavaScript SDK, Former2 will scan across your

infrastructure and generate an output to your liking: CloudFormation, Terraform, CDK Cfn

Primitives/Terraform, and other languages such as Python, Java, C#, & Pulumi - TypeScript, and

finally, even as a draw.io diagram.

I worked previously on a couple of other projects, but really took this project to heart once I saw

its usability. Former2 is actually a fork of an old project used by AWS back in the day, about 10

years ago, so I took it, and continued with it as an open source project.

It can be used via a web interface as well as a command line tool, or you can even host your

own http server.

Former2 does not create any resources within your AWS account and will keep your AWS

credentials locally (in memory) without sending it to the cloud or somewhere else.

You can exclude services, sort outputted data or filter by searching for a specific application.

Former2 takes advantage of the AWS SDK for JavaScript in the browser. It makes Get/List calls

to determine the resources within your AWS account and maps those SDK responses to the

various template formats. Former2 derives its name from the previous CloudFormer tool.

Former2 has evolved to fully support all AWS Services. Both inexperienced and experienced

AWS developers can use Former2 to quickly construct Infrastructure as Code templates for their

companies or customers. You can also add an option to add a deletion policy to your CF

(delete/retain)

What does it cost?

Since it is an open source tool, it will not incur any charges beside the API calls to AWS.

Why open source?

Besides being an awesome concept, as an open source software, anyone can contribute to

Former2, meaning enhancements or bug fixes can be shared by the community. Former2 has

more than 30 pull requests and more than 100 issues raised, which has significantly contributed

to shaping the project. Many of these contributions are raising or fixing bugs that I otherwise

wouldn’t find due to the sheer scale of the AWS products and services. I thank everyone who

has contributed directly or indirectly to the project.

Building trust is also important. Handling AWS credentials is a high-risk activity, and Former2 is

designed to never send these credentials to an external server that isn’t an AWS API endpoint.

Being fully transparent about the code you are running is essential to show that you understand

the sensitivity of the data.

Well, I thank you Ian, for your time, your responsiveness on GitHub and for creating this

awesome solution!

--

Before we start, a couple of words about CloudFormation:

AWS CloudFormation is an Infrastructure as Code (IaC) service, which means you can use text

files (JSON or YAML) to define a set of resources to be deployed without worrying about

semantics of how they are deployed. Using AWS CloudFormation truly helps with consistency

and repeatability.

More information can be found at this location: https://aws.amazon.com/cloudformation/

So what is Former2, actually?

Former2 has a couple of usages, depending on how you look at it:

● It can be a replication tool that you can create or move resources from one environment

to another.

● It can be a backup tool, but not in its traditional sense.

While Former2 provides a current snapshot of your AWS infrastructure, when redeploying in

many services and configurations it may not be relevant the second time. Security groups, local

IPs, AMI IDs or any resources created by Stackset are randomly assigned, and as such, it is not

likely you will get the same ones when restoring. Former2 actually provides you with a list of

resources you had and how they interacted with other services/resources. This precious

information can be a godsend for those who lost access to their account, or if the account

configuration is destroyed by a rough actor, especially if you don’t really know what you had

inside your account.

So, to recap: Former2 provides you with a backup/list of your infrastructure-service connections

and configuration, but not a backup of your data (the content of the EC2 server, for example).

--

While debugging some issues I noticed on Former2, I worked with Ian and we managed to solve

those issues (different behavior on edge scenarios) by communicating via his GitHub account.

This is the power of open source.

While working on the Former2 CLI, I noticed that it takes the AWS user identity from the

~/.aws/credentials file. This means that if you have one user or a couple of local users, you can

use it without any issue.

But what if you are using assume role on the ~/.aws/config file? What if you are getting your

accounts from your organization? With this setup you are stuck, and this great adventure comes

to a screeching halt before you even had a chance to check out this solution.

Fret not, young grasshopper - we have the power of the code.

I looked into the Former2 source code (NodeJs) and found that you can actually use 3 different

options:

1. Using the default configuration in the ‘AWS_SHARED_CREDENTIALS_FILE’ environment

variable.

2. Add a custom profile using the ‘AWS_PROFILE’ environment variable.

3. Directly load the AWS.SharedIniFileCredentials provider.

var creds = new AWS.SharedIniFileCredentials({profile: 'myprofile'});
AWS.config.credentials = creds;

More details can be found at

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SharedIniFileCredentials.html

My use case was a bit more complicated: I had a couple hundred accounts that obviously

cannot and will not be configured with an access key and secret key, so I needed to get

temporary credentials for each account. Next thing was to understand where and how to store

them:

● Where was obvious - on the ~/.aws/credentials file.

● How needed more brainstorming. This is the main file, undoubtedly one of the most

important files, which I can’t use to store hundreds of temporary credentials. It will get real messy, real fast.

Let’s not forget that the default duration of IAM role session is 900 seconds (15 minutes), and

can be up to 12 hours (find more details at

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html).

So in the course of an hour, this file can potentially contain thousands of active redundant

credentials. Searching through this file will become a long and wasteful task, memory wise, and

can pose a security risk if used inappropriately.

So, after putting my thinking cap on, I found a clever solution (if you don’t mind me saying so):

I will loop all my accounts IDs, keep them in a list and request an STS for each account. I will

then rewrite the default credentials file with the default profile and a temporary profile that holds the STS details I received previously as a parameter. The default profile (which holds sensitive data that can’t be placed on Git), will be placed on a vault (by Hashicorp, which is a cool project on its own, but it’s another story for another time) so the details will not be visible at all. This way, the file is nice and clear, dynamic, only contains one assumed credential every time it is called and will not be alive more than a few minutes, until the Former2 finishes its turn. At the

end of the process, the file is cleaned for the next run, and eventually will contain only the

default profile as started.

No messy credentials file, no old details to clean, no info that can be abused.

For those who are wondering what STS means:

STS is a Security Token Service that can provide you, as a user, the ability to assume roles via

API calls. You can use many SDKs to perform the task you need to with either Python, Java,

Node.js, GO, and many more other languages. More information can be found at

https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html

--

Former2 GitHub page: https://github.com/iann0036/former2

Running Former2 CLI:

There are a couple of ways to use this tool, either by installing it from brew.io for mac os users,

as a Docker container or by getting it from a NPM repository.

I used the NPM method. If you don’t already have NPM installed, go ahead and check out your

preferred install method: https://docs.npmjs.com/downloading-and-installing-node-js-and-npm.

To install the CLI tool, run this command:

npm install -g former2

The g flag will install packages globally onto your npm cache folder on your machine.

If you want to update, run this command for the latest version:

npm update -g former2

or

npm update -g former2@0.2.67

for a specific version.

To run the former2 command, you would use (for a single service):

former2 generate --services "EC2" --region "eu-west-1" --profile “profile_name” --output-cloudformation "cfn.yml" --proxy protocol://Host_url:port

Former2 can scan services which are region-specific, globally, or backup all AWS services.

Mind you this can take some time, depending on how many resources you have on each

service.

For information on additional options that can be used, see the help section:

Here is an example of the output: