Following up on my last article about how to evaluate technology options I’d like to take one example and describe how to evaluate cloud mock testing frameworks.
The context is that in my work we had to choose between two popular AWS testing frameworks, namely Localstack1 and Moto2.
I won’t go into the details of the two choices (I leave this for another update), but explain the first step of any evaluation process: making a list of categories that we use to compare the options. This is mostly a brainstorming exercise, with the goal to have a list of attributes that is exhaustive and mutually exclusive.
List of categories
Ease of use
This might be the most important attribute. Because quite often, testing is seen as something useful and required but annoying to do. And I believe the reason for that can be found in something I call developer laziness, which I mean in a positive way (including myself in that category)!
We developers are inherently lazy and that’s why we have an urge to automate mundane tasks as much as possible. This is a subject for another article, what is relevant here is that the if a testing library makes writing tests harder to do, there will be a negative effect on the overall quality of testing.
Ease of use can be subdivided into further attributes:
- does the framework change the way we write tests?
- does it integrate well with our existing tools (e.g.
pytest?) - does it become easier or harder to write tests?
Ease of debugability
I’m unsure if that is actually a word (however, wikipedia3 does list it as a system quality attribute), but it is easy to understand in our example: does the framework help us with debugging our code?
The main purpose of testing is to make sure the code we write does what it is supposed to do. But very often, writing a test has another advantage: it makes debugging simpler (sometimes local testing is even the only way to debug code).
This is certainly true for software that runs on the cloud. It can be difficult to debug code that depends on remote services. Having a local mock simplifies that a lot.
Completeness and Correctness
These attributes are binary in the sense that if a framework doesn’t meet those requirements we can’t use it.
Completeness. With that I mean: does it the framework mock all the services we want to test? If not, we can’t use it.
Correctness. Does the mock behave the same way as the real service? If not, we obviously can’t use the mock.
Speed of execution
The longer testing takes to execute, the more reluctance we have to use it. Long execution time can become a major efficiency problem.
Other categories
- Licensing cost (in our case not relevant as we compare open source frameworks)
- Available support, popularity, long-term availability