Quality attributes of a cloud mock testing framework

Following up on my last article about how to evaluate technology options I’d like to take one example and describe how to evaluate cloud mock testing frameworks.

The context is that in my work we had to choose between two popular AWS testing frameworks, namely Localstack1 and Moto2.

I won’t go into the details of the two choices (I leave this for another update), but explain the first step of any evaluation process: making a list of categories that we use to compare the options. This is mostly a brainstorming exercise, with the goal to have a list of attributes that is exhaustive and mutually exclusive.

List of categories

Ease of use

This might be the most important attribute. Because quite often, testing is seen as something useful and required but annoying to do. And I believe the reason for that can be found in something I call developer laziness, which I mean in a positive way (including myself in that category)!

We developers are inherently lazy and that’s why we have an urge to automate mundane tasks as much as possible. This is a subject for another article, what is relevant here is that the if a testing library makes writing tests harder to do, there will be a negative effect on the overall quality of testing.

Ease of use can be subdivided into further attributes:

  • does the framework change the way we write tests?
  • does it integrate well with our existing tools (e.g. pytest?)
  • does it become easier or harder to write tests?

Ease of debugability

I’m unsure if that is actually a word (however, wikipedia3 does list it as a system quality attribute), but it is easy to understand in our example: does the framework help us with debugging our code?

The main purpose of testing is to make sure the code we write does what it is supposed to do. But very often, writing a test has another advantage: it makes debugging simpler (sometimes local testing is even the only way to debug code).

This is certainly true for software that runs on the cloud. It can be difficult to debug code that depends on remote services. Having a local mock simplifies that a lot.

Completeness and Correctness

These attributes are binary in the sense that if a framework doesn’t meet those requirements we can’t use it.

Completeness. With that I mean: does it the framework mock all the services we want to test? If not, we can’t use it.

Correctness. Does the mock behave the same way as the real service? If not, we obviously can’t use the mock.

Speed of execution

The longer testing takes to execute, the more reluctance we have to use it. Long execution time can become a major efficiency problem.

Other categories

  • Licensing cost (in our case not relevant as we compare open source frameworks)
  • Available support, popularity, long-term availability

How to evaluate technology options

As mentioned before, the answer to the increasing complexity of technology is to provide ever more frameworks, tools and other solutions. Everyone involved in development has to evaluate and choose among options constantly.

Quite often, we use what we know. I call this “developer laziness” and I mean that in a good way. If a tool is too complex to use or understand, it has to provide a substantial advantage to warrant convincing others to invest the time and effort to learn how to use it.

However, not looking beyond what we currently know severely limits the ability to increase the quality of our software implementation. It is especially the role of an architect to find new and better ways to improve the status quo and explore alternatives.

Comparing technology choices is a three step process:

Evaluating technology options

The steps are as follows:

  1. Make an exhaustive and complete list of categories that we can use to compare the features of each option. This list should be MECE1, which is an acronym for mutually exclusive, collectively exhaustive. It’s a binary list of categories that encompass as much area as possible.
  2. With this list in hand, research answers. This includes talking to experts, reading documentation and implementing proof-of-concept mini projects
  3. Fill in the answers, prioritise, weigh categories and then come up with a conclusion.

Step 1) is a brain-storming exercise. I find it very useful to involve other members in this step. If you do so, I recommend having everyone make a list on their own and then merge the findings into a list. Having more than one person come up with categories increase the chances that our list is exhaustive. The final merge of all answers into a final list can be best accomplished by one person to keep the list mutually exclusive.

Step 2) is then best done by the expert of that field. If the technology we explore is new (which quite often it is), implementing a quick proof-of-concept is an effective way to get a “gut feel” for the technology.

Presenting Step 3) is then best done with the team and in front of the stakeholders. If the choice we have to make is substantial, the desired outcome of the process is to give the stakeholder a clear picture of the pros and cons and enable her to make an informed decision.

One question is how much time we should spend on this process. The more important the decision the more thorough we have to be. It is important to make the evaluation of technology itself a task that has resources and time allocated to it. The only way to find a good solution is to thoroughly understand the problem and that requires to give importance to the evaluation process.

I will come back to this process many times with concrete use cases.

Nobody knows but at least someone owns it

Nobody knows what to do.

Admitted, that sounds rather clickbaity, but sometimes being a bit extreme can help make a point.

Technology has become way too complex for one individual to fully comprehend. Any fundamental change will lead to uncertaintenties that make it impossible to forsee every possible outcome.

And we can’t research everything before starting to implement the change we want to have. The time we can invest into reading documentation, interviewing experts and building proof of concepts is limited (mostly but not only because of economical reasons).

We are all guessing, more or less. Some may have a lot of experience in one area, and that certainly helps, but requirements are ever changing as is the environment.

So instead of exactly knowing what to do, we are placing bets. We make (hopefully) good informed decisions and see how they work out.

But what if our bets don’t work out?

Then we correct our assumptions and place another, hopefully improved bet.

To greatly increase the chances that this iterative process leads to a good solution, one concept is essential: it’s called ownership.

Only if someone owns the process of placing bets we can make sure every bet brings us closer to our goal.

If nobody owns the consequence of a decision, we leave it up to chance that someone will take charge to create the improved, modified new bet. However, this new person still doesn’t know.

The following diagram summarizes my point:


What is (software) strategy, anyway?

I kick this newsletter off with a little exploration of the word “strategy”, in particular in the context of software development. It is such a generic word that I think it would be useful to specify what I mean by it. An understanding of something broad and generic is likely to change over time so this is my first attempt but I may come back and refine what I said.

Let me start with what is NOT a strategy:

  • Random actions are not a strategy
  • A plan that is not expressed or formulated in any way is not a strategy.
  • A strategy where it’s key components are not somehow measurable or teachable to others is not a strategy
  • A strategy that can not explain why it leads to certain actions (and not others) is not a strategy
  • A strategy that changes and can’t explain why it does is not a strategy
  • A strategy that allows actions that go against it is not a strategy

So then what is it?


Strategy is a high level plan to achieve one or more goals under conditions of uncertainty. 1

Let’s analyze this definition in the context of software architecture by starting at the end and going backwards.

What are the uncertainties of software?

The most common source of uncertainty stems from entropy. It still amazes me every day how quickly something simingly simple becomes all of a sudden complex and difficult to understand.

Computer professionals answer the challenge of entropy with creating a lot of frameworks, programming languages, design patterns etc. Going one step further, the uncertrainty of software development comes from making the right (technological, design) choice (if there was, for example, only one programming language available (“FORTRAN”), then there would be no uncertrainty in choosing it).

What are goals?

The goal of software can vary a lot depending on the context where it is used (e.g. open-source versus comercial software). In this newsletter, I solely focus on developing comercial software. That means, the primary goal is to meet business goals. Without a functioning business, there is no software. Does the design and architecture of our software ensure that it delivers everything necessary to succeed as a business?

Out of the business goals we can derive functional and non-functional requirements. Being an architect, I write mostly about non-functional requirements or goals

What is a high-level plan?

Now the final missing piece of the definition: what do we mean with a “high-level plan” in terms of software architecture?

“Having a plan” is what most people would think of what makes a strategy.

In software development, a plan consists of three ingredients:

  1. Specification of requirements
  2. Research of options and choosing among the options
  3. Proposing actions that meet requirements and are based on the choices made in step 2)

Requirements can be functional and non-functional, options are technological and architectural design choices and actions are concrete steps to implement software the meets the requirements.

Software Strategy

Depicted what was said we get following diagram:

Software Strategy

Business goals define requirements. Entropy and uncertainties motivate creation of options (technological, design, execution) and choosing from those options lead to actions. Strategic planning (requirements + options + actions) helps to meet business goals.


Taking everything together, I use the word strategy in the context of software development as follows: a strategy consist of strategic planning which is a three step process of requirements gathering, evaluation of options and specification of actions in order to meet business goals in an environment of increasing entropy.

Using Clojure on AWS Lambda

Summary: Dive into serverless with Clojure. We explore ways to use Clojure instead of Java to implement AWS Lambda functions. Following the official AWS documentation, we first implement “Hello Worlds” using Java which we then emulate in Clojure. Pure Java-only developers can skip the Clojure part, and Clojure developers may learn more about Java interop.

What is AWS Lambda?

AWS is a serverless computing platform that runs code in response to events and automatically manages the computing resources that the execution of the code requires 1. In other words, with AWS Lambda you can execute Clojure (or other languages) code without the hassle of setting up a server environment. Just assemble and upload a JAR file of your code, and you are ready to go.

The word “serverless” is a bit of a misnomer. There are still servers running in the background, but they are entirely hidden.


You are going to need an account at AWS and have the AWS CLI tool installed. Please refer to the official documentation on how to do that.

Source code

Available as usual on Github2. Clone that or follow along coding everything yourself. The easiest way is to use leiningen new clojure-aws-lambda with two source folders: src/clojure and src/java.

Configure permissions

Before we can execute Lambda functions, we need to set up the permissions and create a role using IAM. Following JSON grants a basic trust relationship with our lambda functions.

  "Version": "2012-10-17",
  "Statement": [
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      "Action": "sts:AssumeRole"

Using the AWS CLI we can create a new role:

aws iam create-role \
--role-name basic_lambda_role \
--assume-role-policy-document fileb://resources/trust_relationship.json

Observe the response:

ROLE    arn:aws:iam::248689245702:role/basic_lambda_role    2018-10-01T14:39:50Z    /    AROAISQSNKXPE4G4F4GFE    basic_lambda_role
STATEMENT    sts:AssumeRole    Allow    
PRINCIPAL    lambda.amazonaws.com

AWS returns the ID of the role arn:aws:iam::248689245702:role/basic_lambda_role. Keep note or use aws iam list-roles to retrieve it later.

Next, we need to attach a policy (we use default policy provided by AWS, the ID is arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole)

aws iam attach-role-policy \
--policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole \
--role-name basic_lambda_role

Implement Lambda function handler

The way to run Clojure on Lambda is to emulate the Java approach.

AWS supports two ways for creating a handler on the Java platform3:

  1. Implement the function method directly using plain Java 4 without using an interface.
  2. Implement the function by leveraging a predefined interface (RequestHandler or RequestStreamHandler)5.

1. Implement directly without using an interface

1.1 Using plain Java

public class PlainJavaHandler {
    public String myHandler(String input, Context context) {
        return String.valueOf("Hello " + input.get("firstName") + " " + input.get("lastName"));

To create a new Lambda function, we need to build our project and upload a jar file containing all the dependencies. We are using Leiningen, the Clojure build tool, to compile all the Java and Clojure code.

To create the jar file, run lein uberjar now.

This creates a new jar file target/clojure-aws-lambda-0.1.0-SNAPSHOT-standalone.jar which we can upload to AWS. (Note: this jar contains the code for the handler for all of the Java and Clojure cases. In a real project, we would not pack everything into one jar file; we do it here for the sake of simplicity).

Create function using the AWS CLI (for a list of commands see 6)

aws lambda create-function \
--function-name PlainJavaHandler \
--runtime java8 \
--role arn:aws:iam::248689245702:role/basic_lambda_role \
--handler javahandler.PlainJavaHandler::myHandler \
--zip-file fileb://target/clojure-aws-lambda-0.1.0-SNAPSHOT-standalone.jar \
--region us-east-1

The name of the handler follows this schema: <package>::<method>

Note: use the proper values for region and add --profile <NAME> in case you have created a dedicated user for lambda.

We are going to test this function using the following input:


We can use the invoke command of the CLI to send the content of this file to the lambda function.

aws lambda invoke \
--function-name PlainJavaHandler \
--payload fileb://resources/string-input.txt \

The response is captured in out.txt.

$ cat out.txt
"Hello World!"

It works.

Direct approach Clojure version

Now let’s implement the same handler in Clojure. The Clojure code has to generate a class file with similar properties.

Ahead of time compiling

Clojure compiles all code on the fly into JVM bytecode which will be loaded dynamically. However, to execute the code on Lambda platform, we need to enable ahead-of-time compiling 7, so we can use the gen-class macro 8 to create the Java classes from our Clojure code.

We enable AOT in our project definition using the :aot :all keywords:

(defproject clojure-aws-lambda "0.1.0-SNAPSHOT"
  :source-paths ["src/clojure"]
  :java-source-paths ["src/java"]
  :dependencies [[org.clojure/clojure "1.8.0"]
                 [com.amazonaws/aws-lambda-java-core "1.1.0"]]
  :aot :all)

Now we can implement the Clojure code:

(ns clojurehandler.plain_clojure_handler
    :name "clojurehandler.PlainClojureHandler"
    :methods [[myhandler [String com.amazonaws.services.lambda.runtime.Context] String]]))

(defn -myhandler [this s ctx]
  (str "Hello" s "!"))

Notice the function signature. We need to pass in a self-reference this due to how the Clojure compiler generates the Java classes.

We can now build and deploy the jar and create a new lambda function which we name PlainClojureHandler:

aws lambda create-function \
--function-name PlainClojureHandler \
--runtime java8 \
--memory-size 256 \
--role arn:aws:iam::248689245702:role/basic_lambda_role \
--handler clojurehandler.PlainClojureHandler::myhandler \
--zip-file fileb://target/clojure-aws-lambda-0.1.0-SNAPSHOT-standalone.jar \
--region us-east-1

Note: To make this work, I needed to increase the memory-size of the lambda function to 256 MB (the default memory size is 128mb, which leads to a java.lang.OutOfMemoryError: Metaspace error).

Now we can invoke the function using our test file that contains the simple string:

aws lambda invoke \
--function-name PlainClojureHandler \
--payload fileb://resources/string-input.txt \

The response is again captured in out.txt

$ cat out.txt
"Hello World!"

1.2 Implement using POJOs

Instead of using primitive input, we can use a POJO as request and response parameter.

package javahandler;
public class Request {
    private String firstName;
    private String lastName;
    public Request(){}
    public Request(String firstName, String lastName) {
        this.firstName = firstName;
        this.lastName = lastName;
    /** ... getter and setter omitted ... **/
package javahandler;

public class Response {
    private String hello;
    public Response(){}
    public Response(String hello) {
        this.hello = hello;
    /** getter and setter omitted.... **/

Now our handler function uses POJO classes in the method signature: Response myHandler(Request input, Context context):

package javahandler;

import com.amazonaws.services.lambda.runtime.Context;

public class POJOJavaHandler {
    public Response myHandler(Request input, Context context) {
        final Response response = new Response();
        response.setHello("Hello " + input.getFirstName() + " " + input.getLastName() +"!");
        return response;

With this in place, we can send JSON input to our lambda function that AWS will automatically (de-)serialise:

  "firstName": "John",
  "lastName": "Smith"

Creating a new function that we call POJOHJavaHandler (just replace --function-name and --handler:

aws lambda create-function \
--function-name POJOJavaHandler \
--runtime java8 \
--role arn:aws:iam::248689245702:role/basic_lambda_role \
--handler javahandler.POJOJavaHandler::myHandler \
--zip-file fileb://target/clojure-aws-lambda-0.1.0-SNAPSHOT-standalone.jar \
--region us-east-1

Let’s test this function, sending the JSON to the function:

aws lambda invoke \
--function-name POJOJavaHandler \
--payload fileb://resources/test-input.json \

Have a look at the response:

$ cat out.txt
{"hello":"Hello John Smith!"}

Note: we could have used JSON in the plain Java approach as well. However, AWS does not serialise JSON into a String. Instead, the input will be a Map<String, Object>. As an exercise, you might want to try to modify the plain Java version and send the JSON file as input.

POJO Clojure version

The natural choice in Clojure to implement a POJO class should be using a defrecord9.

won't work
(ns clojurehandler.pojo_clojure_handler)

(defrecord Request [firstname lastname])
(defrecord Response [hello])

  :name "clojurehandler.POJOClojureHandler"
  :methods [[myhandler [clojurehandler.pojo_clojure_handler.Request com.amazonaws.services.lambda.runtime.Context] clojurehandler.pojo_clojure_handler.Response]])

(defn -myhandler [this request ctx]
  (Response. (str "Hello " (:firstname request) " " (:lastname request) "!")))

Unfortunately, though this compiles it won’t work. AWS uses Jackson to serialise JSON requests which requires the empty default constructor of our Request and Response classes. In the immutable world of Clojure, a defrecord won’t generate such constructor for us. We can’t use them.

We could use gen-class to generate the POJOs. However, that would feel awkward. We’d have to create getters and setters and so on. Instead, we can “cheat” here and use the Java POJOs that we have already defined. This does make sense as POJOs don’t belong to Clojure’s world.

(ns clojurehandler.pojo_clojure_handler
  (:import (javahandler Request Response)))

  :name "clojurehandler.POJOClojureHandler"
  :methods [[myhandler [javahandler.Request com.amazonaws.services.lambda.runtime.Context] javahandler.Response]])

(defn -myhandler [this request ctx]
  (Response. (str "Hello " (.getFirstName request) " " (.getLastName request) "!")))

(Note: I still had to fully qualify the Request and Response classes in gen-class, thus actually not using the import of Request.)

Let’s create the function (remember to provide extra memory):

aws lambda create-function \
--function-name POJOClojureHandler \
--runtime java8 \
--memory 256 \
--role arn:aws:iam::248689245702:role/basic_lambda_role \
--handler clojurehandler.POJOClojureHandler::myhandler \
--zip-file fileb://target/clojure-aws-lambda-0.1.0-SNAPSHOT-standalone.jar \
--region us-east-1
aws lambda invoke \
--function-name POJOClojureHandler \
--payload fileb://resources/test-input.json \

Have a look at the output to confirm it worked:

$ cat out.txt 
{"hello":"Hello John Smith!"}

2. Implement a predefined interface

2.1 RequestHandler interface

Instead of relying on the input and output parameter of our handler method, we can leverage the RequestHandler interface of the AWS sdk:

package javahandler;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
public class RHJavaHandler implements RequestHandler<Request, Response> {
    public Response handleRequest(Request input, Context context) {
        final Response response = new Response();
        response.setHello("Hello " + input.getFirstName() + " " + input.getLastName() +"!");
        return response;

This does the same as the Java POJO example from above. When creating this function, we only need to specify the class name and can omit the function name: javahandler.RHJavanHandler

aws lambda create-function \
--function-name RHJavaHandler \
--runtime java8 \
--role arn:aws:iam::248689245702:role/basic_lambda_role \
--handler javahandler.RHJavaHandler \
--zip-file fileb://target/clojure-aws-lambda-0.1.0-SNAPSHOT-standalone.jar \
--region us-east-1

I’m skipping the test, re-use the command from above replacing the function name.

RequestHandler Clojure version

The straightforward solution in Clojure should be using the gen-class macro to implement the RequestHandler interface. Like this:

Won't work!
(ns clojurehandler.rh-clojure-handler
  (:import (javahandler Request Response)))
  :name "clojurehandler.RHClojureHandler"
  :implements [com.amazonaws.services.lambda.runtime.RequestHandler])
(defn -handleRequest [this request ctx]
  (Response. (str "Hello " (.getFirstName request) " " (.getLastName request) "!")))

Unfortunately, at the time of writing (and to my knowledge) Clojure does not handle concrete type parameter of generic interfaces. Though the code compiles fine, deploying and running results in following error message: Class clojurehandler.RHClojureHandler does not implement RequestHandler with concrete type parameter.

It seems we can’t follow this approach with Clojure. We could provide an implementation in Java with concrete type parameters and use that in Clojure, but this becomes again somewhat awkward (even more since solution 1.2 works in Clojure, we don’t need that interface). Let’s move on to the final approach, taking advantage of streaming.

2.2 RequestStreamHandler interface

package javahandler;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestStreamHandler;import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.charset.StandardCharsets;
import java.util.Scanner;

public class RSHJavaHandler implements RequestStreamHandler {
    public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context) throws IOException {
        String name;
        try (Scanner scanner = new Scanner(inputStream, StandardCharsets.UTF_8.name())) {            name = scanner.useDelimiter("\\A").next();
        outputStream.write(("Hello " + name + "!").getBytes());

This is the most flexible approach. We need to convert the inputstream and then write to the outputstream. Test this by sending the test input containing the string (string-input.txt, though this will contain the quotationmarks in the result).

If we want to send JSON we need to do the deserialisation in the function method.

RequestStreamHandler Clojure version

The Clojure version can be implemented straight away without issues:

(ns clojurehandler.rsh_clojure_handler
  (:require [clojure.java.io :as io]))
  :name "clojurehandler.RSHClojureHandler"
  :implements [com.amazonaws.services.lambda.runtime.RequestStreamHandler])
(defn -handleRequest [this input output ctx]
  (let [name (slurp input)]
    (with-open [o (io/writer output)]
      (.write o (str "Hello " name "!")))))


Using Clojure on AWS Lambda has its quirks. As demonstrated above, it is not always a straightforward process to replace Java with Clojure. The lacking support of concrete type parameters for generic interfaces and the immutability of types require some workarounds to use Clojure for implementing Lambda functions. The best ways are either implementing the RequestStreamHandler (2.2) or using the direct approach with standard (1.1) or custom types (1.2).

Taking all together, it feels slightly forced to use Clojure, making it a valid point to say that it might not (yet!) be the time to use it on AWS Lambda. Even less so that it seems necessary to allocate additional memory to execute the bytecode generated by the Clojure compiler, generating additional cost.

For now, a more practical approach is to implement Lambda functions in Java and keep the backend in Clojure.