Serverless Ingestion of Caliper Events to an AWS SQS Queue

The IMS Caliper Analytics® Specification ``provides a structured approach to describing, collecting and exchanging learning activity data'' (Caliper Analytics® Specification, version 1.1).

It also defines a method for transferring the data from a learning environment to an endpoint for storage. In a nutshell, learning activity data are sent via an HTTP POST to an endpoint with authentication provided by a bearer token.

Thus any web server and language can be used to receive the learning data and save it to storage or a database. Using various AWS services this can be achieved without a web server.

As an example, the following describes how to create an application to receive data from a Caliper Sensor and send it to an SQS queue for collection and processing by another application. Instead of SQS, the application could just as easily save the data to DynamoDB or S3.

The application will receive the data via https and use a custom domain name.

The AWS services used are:

API Gateway (Networking & Content Delivery)
This receives the HTTP POST (Method Request) from the client, does some transformation on it and passes it to Lambda (Integration Request). It then receives the response from Lambda does some transformation on it (Integration Response) and makes the HTTP response to the client (Method Response).
This is also where we associate a custom domain with the application.
Lambda (Compute)
This processes the request. It checks the bearer token and sends the learning data to the SQS queue. This is where the programming code goes (Lambda function).
SQS (Application Integration)
This is where we create the queue that will receive the data.
IAM (Security, Identity & Compliance)
A policy and a role are defined here to allow our Lambda function to write to the SQS queue.
Certificate Manager (Security, Identity & Compliance)
This is where we get an SSL certificate for the domain.

Some of the services have dependencies on the others.

Thus the order they are shown here is perhaps not the order you would naturally start with, but it save having to start in one area, go elsewhere, and then return.

First, create the SQS queue and the HTTPS certificate. These do not depend on anything else. The certificate is not needed until the end, but it can take a little while to be verified, so create it now.

The IAM role will be used by the Lambda function and its policy needs to know the SQS queue ARN. So create the role and policy now.