top of page

SQS or API Gateway?

Updated: Sep 30, 2022


A while ago, I was handed out a task where I was required to get data from an external source, process and parse it using a Lambda function which will, in turn, redirect said data to the needed AWS services.

Upon pondering the situation, the easiest way possible to get this task done is to use SQS, which is short for ‘Simple Queue Service’, (details here). SQS is an excellent service created by AWS that provides a queueing service; obviously, you can use other services like, RabbitMQ or NATS, but then you would have to manage it yourself and that is an overhead you might not need depending on the scale of your operation, available manpower, and let’s not forget – the time to go ahead and actually create an awesome solution for you or your customer.

As the name suggests, it is a simple service, so simple that I could have finished the task in 5 minutes, consider myself a hero and spend the rest of the remaining time of the task looking at motorcycles vids on YouTube, Ahem – I meant, studying for my upcoming AWS certification exam. (-; But seriously, where is the fun in that? Let’s learn and expand our knowledge with AWS tools and services to find another way to handle incoming data. Let’s use API Gateway!


API Gateway is a powerful tool that allows you to have a managed service without the hassle of setting, configuring, securing and managing it. From AWS website: API Gateway handles all the tasks involved in accepting and processing up to hundreds of thousands of concurrent API calls, including traffic management, CORS support, authorization and access control, throttling, monitoring, and API version management. It has no minimum fees or startup costs.”


How API Gateway can be used:


Well, let’s get down to business.

Since we’ll be using STS for enhanced security, we do not have any static Access Key Id & Secret Access Key but a temporary token when assuming a role that will allow us to connect to different AWS services. But, there is a catch… The requests Python package uses 2 variables when using HTTPBasicAuth, Whereas STS provides 3 variables.. So what to do? Are we doomed to go back to our SQS option? Well, not in the least!

After some digging, I found two GitHub repositories that seem to be helpful: ‘aws-sigv4-lambda’ & requests-aws4auth. In the end, I opted to use requests-aws4auth, since it was much relevant to my use case. We will use the requests-aws4auth as the authorization layer when using the requests package. This package will also assist us in using the AWS_IAM configuration inside our API gateway, as discussed later on in this guide.

Initial installation is pretty straight forward, just run `pip install requests-aws4auth`, and you are ready to play. If you encounter any issues, run these commands: `python -m pip install --user --upgrade setuptools wheel` # (use python3 if you’re using python 3) `python -m pip install --user --upgrade twine` Finally, make sure the package is installed using: `pip freeze | grep requests-aws4auth`


API Gateway Configuration:

On the left side pane of the service page you can find many options for configuration, let’s go over them and see what we can do with them:



















Resources will show you the current methods you have created, as well as the authorization method and API Key. Stages: you can use this as an environment like development, staging or production, but you can call it anything you want really. Authorizers: for API access control you can use Cognito User Pools or a Lambda function. Gateway response: anything from Access Denied to WAF Filtered, this is a semi-canned response that you can use. Note that Proxy integrations cannot be configured to transform responses. Models: this is actually the API schema, or more simply put, this is how you call the API. Besides the two preconfigured API schema for an empty or an erroneous request, you would need to add your own, in this case, I have created this module: Hint: You can go to https://json-schema.org to learn more about API schema.

{ "$schema": "http://json-schema.org/draft-04/schema#", "type": "object", "properties": { "accountid": { "type": "string" }, "priority": { "type": "string" } } }


Resource Policy: this is a security measure that will enable you to further tighten your client’s access to this API, either by account, Source VPC, VPC Endpoints, and/or IP range. This policy is in the form of IAM policy. Documentation: you can create an extensive documentation for your API or import from a Swagger file. Dashboard: Monitor your API. Settings: Where you can further configure your API, including your API endpoint name - you can use a custom domain name to provide a more friendly approach for your users. Usage plan: You can create a usage plan for your API that includes rate limiter, burst limiter, and quota; you can also connect your API to a product registered with the AWS SaaS Marketplace. Now, here is a nice little trick that amazon probably didn’t intended for us to use.. Intentionally the api key should be used to verify the usage plan, but we can use it in our advantage to tighten our security, so only the one who have this api key will be able to call it! How cool is that (-: ? For it to work, you need to create a Usage plan: press the blue button ‘Create’, then Create an API Key and attach it to the correct plan. Eventually the usage plan will be configured with the API in question, the stage (dev, etc), the method (Post, etc) and the throttling configuration. Client Certificates: use client certificates to verify the requester's authenticity. Settings: the last settings – this is for CloudWatch logs: enabling logs for your API is helpful for debugging, but it can dramatically slow down any response you get or send, so actually using logs is not recommended.


What you need to know about.. Proxy integration: Using proxy integration is quite helpful when utilizing a Lambda function; it will provide much more relevant details in the event than if you don’t. Your Lambda function will have full control on the payload it sends back to the requester, note that the output from Lambda needs to be in certain format, For example: response = { 'statusCode': 200, # or any other code, 400, 500, etc. according to the condition inside your Lambda. 'body': You are Awesome', # what the requestor will get when calling the API. 'headers': { 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': '*' } }

AWS_IAM, Signature Version 4: When using AWS_IAM as the authorization type for your API method, the requests need to be signed with Signature Version 4 signing process. It is not necessary to use resource policy on top of that, unless you want to limit access down to certain IP range. In order to sign a request with Signature Version 4, a set of AWS credential is required. The signature will be sent to AWS API endpoint, and it will be used to determine if the credential belongs to someone with the proper permission to carry out the requested action. The credential consists (usually) of an access key and a secret access key. In Python, the easiest way to sign your request is via a library called “requests-aws4auth as mentioned above.


Final look at the method configured:

* One last thing that you might think is silly, but great man have already fallen due to this mishap (Don’t ask me how I know),Is to never forget to deploy the changes you’re performing to the API configuration. Otherwise, it will not be reflected when you’ll test it. To do so, go to Actions at main page and select ‘Deploy API’:


Code:

Here is a full code example for triggering the API Gateway:

payload = {

"accountid":accountid,

"priority":priority

}


def get_sts():

client = boto3.client('sts')

response = client.assume_role(RoleArn='arn:aws:iam::<accountid>:role/<AssumedRoleName>',RoleSessionName='ApiGatewayInovocation')

aki = (response['Credentials']['AccessKeyId'])

sak = (response['Credentials']['SecretAccessKey'])

st = (response['Credentials']['SessionToken'])


def access():

sts = get_sts()

AccessKeyId = (sts['aki'])

SecretAccessKey = (sts['sak'])

SessionToken = (sts['st'])

headers = {'content-type': 'application/json','Accept': 'application/json','x-api-key':'mamwLC21at8bIa1T0FdFo3HBfah1'}

auth = AWS4Auth('{}', '{}', 'us-east-1', 'apigateway',session_token='{}'.format(AccessKeyId,SecretAccessKey,SessionToken))

apigw = requests.post("https://abcd1234.execute-api.us-east-1.amazonaws.com/test", data=json.dumps(payload), headers=headers, auth=auth)

print(apigw.text)


Examples of the complete Signature Version 4 signing process (Python) https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html


Creating API Gateway via console:

Press the ‘Create API’ button on the right.



At the next page, you will see that there are 4 options to choose from: HTTP API, WebSocket API, REST API, and REST API Private. Every API type has its own unique possibilities and powers - for our use case, we’ll go ahead and choose ‘REST API’.


Once you’ve selected the API type you need, on this page you would need to decide about the resources & method the API will use – as well as the naming and endpoint type.


Now, you will need to configure the HTTP method you will be using, in our case, it will be POST: Creating the method is easy – go to ‘Actions’ > ‘Create Method’ > Select the Method from the drop-down list. Select the integration type, whether to use Proxy integration, region and lambda function (if you’re using one). Using proxy integration is quite helpful when utilizing a Lambda function; it will provide much more relevant details in the Lambda event than if you don’t.


After configuring the whole process, this is the final flow for our API would look like- you can see the client - which triggers the API, what method is used, transferred via the proxy to the Lambda function (or not), then return a response with the result of your request. Note the little orange\blue dot – it will lead you to more configurable options for your API, and provide more hints during the configuration process. Once done, you can press the test button to test your API and see the response. You can also see results in CloudWatch log groups that is attached to your Lambda function.





bottom of page