Functional Approaches - Reader, Algebra and Interpreter

I have developed in Scala since 2015 and the current project in which I am assigned to found a big application that has 4+ years of development. Sadly, because of multiple reasons (one of the first projects developed in Scala in the company, developers that were learning the language and developing the current product at the same time, developers that tried to do Imperative Scala or Java in Scala and so on) we have several microservices that takes little or no advantage from the functional capabilities of Scala.

Recently we needed to create a new microservice and I am one of the main developers, so, because of its nature I have decided to put into practice some concepts that I learned from the excellent Functional and Reactive Domain Modelling by Debasish Ghosh. Also I have seen this opportunity a new way to teach ever member of my team how can we make well-designed, maintainable and readable software.

So, let's go!

Before we start...

I created a toy project called ReaderM that you can find here. It consists of a very basic API for managing my music tastes: classify my favourite artists, albums, bands and songs. In this stage of development you can see it as a very basic CRUD application with operations for querying, storing and updating data. The current implementation is made only for the Artists entity and it will be extended to the rest of the entities of the model.

If you're asking yourself: why that model/use case? It's because of my lack of imagination :sweat_smile:. I didn't wanted to reuse the same examples (inventory, coffee shop, online store and so on). It is a pretty straightforward model and I hope to illustrate the concepts of this post with it.

You can clone the project and follow the instructions of the readme for setting up the application.

One last thing about this: the set of technologies chosen for this project will let me write another couple of posts demonstrating its usage, but, for the scope of this post let's concentrate on some patterns that I'll like to introduce here.

From Algebras and Interpreters...

Debasish, in his book, talks about the importance of an "Algebraic" design and how we can gain three characteristics in our application:

• Loud and clear design.
• Compositionality.
• Verifiability.

Citing the section 3.1 of his book:

A module, as defined in a functional domain model, is a collection of functions that operates on a set of types and honor a set of invariants known as the algebra of the module. In mathematical terms, this is known as the algebra of the module.

In this case, the algebra is our published contract of the domain that we are modeling. This contract exposes the operations related to our domain model. In our case, we expose the algebra of the operations that we can do with our Artists.

We'll see more details shortly.

... To Reader

Reader monad is not new. We'll explain it in a moment, but first, I like to reference a couple of excellent talks by Rúnar Bjarnason about it and its usage for Dependency Injection:

• Lambda: The Ultimate Dependency Injection Framework
• Dead-Simple Dependency Injection

I strongly encourage you to take a look to these talks. These are pretty short (~30 mins) and explain how to use this monad for the purpose that I've used it here.

The Algebra

Our Artist domain is composed of certain operations in which we can rely on to manage our aggregate. Its published contract is as follows:

trait ArtistService[T] {

  def listArtists(): Reader[ArtistRepository, List[T]]
  def retrieveArtistById(id: Int): Reader[ArtistRepository, Option[T]]
  def retrieveArtistByName(name: String): Reader[ArtistRepository, Option[T]]
  def insertArtist(artist: T): Reader[ArtistRepository, Int]
  def updateArtist(artist: T): Reader[ArtistRepository, Int]

}

Let's take a detailed look at certain characteristics of this algebra:

1. Trait parametrization: you can see that our algebra is parametrized on T type. It means that the concrete implementation of this service will need to explicitely provide a type that suites the definition of every operation. In our case there's only one, but it could be more, depending of the needs of your algebra. What can we gain with this? Basically two things: making our contract explicit and preventing leakages. About the last statement: our implementation needs to attach to a set of invariants, no matter the concrete implementation.
2. Compositionality: you can add new operations to your algebra that could be implemented in terms of existing operations. For example: I can implement insertArtist looking for an Artist by name and if I couldn't find it then create it. For that purpose I can use the mentioned operation with retrieveArtistByName. I'm composing the first operation taking advantage of the latter.
3. Abstraction over evaluation: one thing that we tend to think when we don't have functional background is that every operation must produce a value. In FP, we appreciate to have an abstraction and rationalize in terms of it. That is, if listArtists "returns" a Reader[ArtistRepository, List[T]], I could use that monadic context for composing other operations that returns the same container. I don't have the value that listArtists is intended to produce (List[T]), but instead I have an abstraction that is way more powerful. More on this later.

As you can see, we have an Algebra that explicitely defines what its operations are, what their evaluation types are and in which types it relies on.

Using types that can by composed (a.k.a Monads) like Try, Future, Option, Either and so on you can compose these "small operations" to form larger operations or programs that are evaluated for producing a value. That is one of the main points for having an algebra and it's the core of the abstraction over evaluation.

Let's see that with an example: suppose that when I use insertArtist the operation returns an Int with the internal ID assigned in the database to that artist and I want to make an operation that, if the insertion succeeds returns the artist inserted. Please detail the return types of these two operations:

def insertArtist(artist: T): Reader[ArtistRepository, Int]
def retrieveArtistById(id: Int): Reader[ArtistRepository, Option[T]]

We have a monad (Reader) that can be composed; also we have two operations potentially composable, because, in the order of actions that we need to execute, the evaluation of insertArtist produces an Int (the artist ID assigned in the DB) that can be used as the parameter id for the retrieveArtistById operation that finally yields an Option[T] with the artist information when evaluated.

So, our new operation will look like this:

def insertAndRetrieve(artist: T): Reader[ArtistRepository, Option[T]] = {
    for {
      id     <- insertArtist(artist)
      artist <- retrieveArtistById(id)
    }
      yield artist
  }

Please note the return type: another reader! Can you see the power of composing abstractions instead of evaluating them?

That function is a program composed by another two programs in itself. We don't need to "evaluate" the result of insertArtist for getting the ID and then calling retrieveArtistById with the ID, thing that you normally do imperatively. In our case, we're totally following the types and composing these bigger implementations based on the primitive ones.

Finally, you need to know that your algebra MUST reflect the published contract for the domain (or part of the domain) that you are encoding there. These are your principal domain operations, you are not publishing auxiliary operations like, for example, traversing a list of domain objects for applying some transformation. That is not part of your domain, even if one of the steps of the domain operations needs to do that.

How to interpret the algebra?

Your algebra is a contract, it is not a concrete implementation. A truly advantage of that fact is that you can create different implementations of the same contract. That is, you need to give an implementation to the operations of your domain and you can do it depending on the context.

In this case, our ArtistService has an ArtistServiceInterpreter (located in the package io.github.alejandrome.algebra.interpreter) that has the following implementation:

class ArtistServiceInterpreter extends ArtistService[Artist]{

  override def listArtists(): Reader[ArtistRepository, List[Artist]] = Reader {
    repo: ArtistRepository =>
      repo.query()
  }

  override def retrieveArtistById(id: Int): Reader[ArtistRepository, Option[Artist]] = Reader {
    repo: ArtistRepository =>
      repo.query(id)
  }

  override def retrieveArtistByName(name: String): Reader[ArtistRepository, Option[Artist]] = Reader {
    repo: ArtistRepository =>
      repo.query(name)
  }

  override def insertArtist(artist: Artist): Reader[ArtistRepository, Int] = Reader {
    repo: ArtistRepository =>
      repo.insert(artist)
  }

  override def updateArtist(artist: Artist): Reader[ArtistRepository, Int] = Reader {
    repo: ArtistRepository =>
      repo.update(artist)
  }

}

object ArtistServiceInterpreter extends ArtistServiceInterpreter

Let's take a detailed look:

1. Implementation as a class: we have a class that extends from our algebra, and in this case we are providing the necessary type (Artist) to materialize every operation that we need to implement.
2. override modifier: meaning that we are implementing the operations specified by the algebra.
3. Interpreter materialization: the last line of our interpreter is a way to create a single instance of our interpreter for using it where needed.
4. Implementation: each function has its own implementation, in this case using the provided Reader parameters to produce an output. More of this later.

As you can see, that is a straightforward implementation.

Note that there are some operations (like retrieveArtistById) that have primitive types provided in advance. We can replace them and provide generic type parameters for them if we want to completely seal our implementation to work with explicitely specified types.

Repository pattern

As you noted, our application relies on persisting data in a Database. Normally we would create some DTO's and then code the persistance layer to interact with the DB. In this case, we're persisting an entire domain entity, instead of mixed domains, sometimes represented by a DTO. This pattern has a whole chapter in Domain-Driven Design: Tackling Complexity in the Heart of Software and represents one of the pilars of DDD.

According to P of EAA catalog by Martin Fowler, the repository pattern:

Mediates between the domain and data mapping layers using a collection-like interface for accessing domain objects.

It is extremely important to understand that the repository (in this case the database) give us a way to persist things, but the storage model in it doesn't reflect our domain model. A co-worker of mine said it recently and he's damn right:

Today's mantra: the database model is not the domain model (no matter how hard ActiveRecord tries to tempt you).

— David Castillo (@castillobgr) June 30, 2017

The whole idea of this pattern is to give a mechanism to map between the domain and the persistence, without messing with our aggregates and entities.

I'm citing (verbatim) the advantages of this pattern, explained by Eric Evans:

• They present clients with a simple model for obtaining persistent objects and managing their life cycle.
• They decouple application and domain design from persistence technology, multiple database strategies or even multiple data sources.
• They communicate design decisions about object access.
• They allow easy substitution of a dummy implementation for use in testing (typically using an in-memory collection).

Finally, have you heard of the Object-relational impedance mismatch? Well, this is a personal opinion, but that's why ORM's are a no-no for me. In this case, you'll see how the Repo. pattern will help you to ease the pain of relying on one of those things.

It's not bad to take a look to a very controversial opinion about it.

Implementation of a Repository

We have the following algebra defining a very simple repository:

trait Repository[A, Id] {
  def query(id: Id): Option[A]
  def insert(obj: A): Int
  def update(obj: A): Int
}

You can see that this is a published algebra for any repository that we want to implement. It has the very basic operations on our domain object (Artist): query by id, insert or update an entire artist. Note that the semantics of this trait are totally generic, the "Artist" part is a way more concrete.

Now let's specialize this contract a little bit:

trait ArtistRepository extends Repository[Artist, Int]{
  def query(): List[Artist]
  def query(id: Int): Option[Artist]
  def query(name: String): Option[Artist]
  def insert(obj: Artist): Int
  def update(obj: Artist): Int
}

As you can see, ArtistRepository is a more specialized algebra that implements the basic operations of a repository but also adds its own concerning operations that aren't necessarily shared with other domain objects. Having said that, you can also implement an AlbumRepository, BandsRepository and SongsRepository based on Repository and each one of them could have their own operations, with the sole condition of implementing the very basic operations published by the generic contract.

In this case, ArtistRepository is offering you two more ways of querying: list all the table, giving you a list of domain objects or seeking an object by the artistName. Again, note the parametrization on types.

Now, let's wonder something: How many implementations an ArtistRepository could have?

The answer is: as many as the number of RDBMS/NoSQL engines we need to provide support for!

This is another huge advantage: provide an interpretation for a repository for each possible concrete database engine we need to support. With that in mind, we can swap our repository implementation easily without needing to make any kind of change to dependent layers. Even for testing, we can provide a dummy or mocked repository for our unit tests very easily.

Let's see how does it looks like our interpreter for PostgreSQL:

class ArtistPGRepository extends ArtistRepository{

  import io.github.alejandrome.config.ApplicationConfig._

  def getTransactor[T](q: ConnectionIO[T]): IOLite[T] = {
    for {
      connection     <- HikariTransactor[IOLite](driverClassName, connectionString, userName, password)
      _              <- connection.configure(hx => IOLite.primitive(hx.setAutoCommit(true)))
      databaseAccess <- q.transact(connection) ensuring connection.shutdown
    } yield databaseAccess
  }

  override def query(): List[Artist] = {
    val q = sql"SELECT * FROM ARTISTS".query[Artist].process.list
    val tx: IOLite[List[Artist]] = getTransactor(q)
    tx.unsafePerformIO
  }

  override def query(id: Int): Option[Artist] = {
    val q = sql"SELECT * FROM ARTISTS WHERE ARTISTID = $id".query[Artist].option
    val tx: IOLite[Option[Artist]] = getTransactor(q)
    tx.unsafePerformIO
  }

  override def query(name: String): Option[Artist] = {
    val q = sql"SELECT * FROM ARTISTS WHERE STAGENAME = ${name.toUpperCase()}".query[Artist].option
    val tx: IOLite[Option[Artist]] = getTransactor(q)
    tx.unsafePerformIO
  }

  override def insert(obj: Artist): Int = {
    val q = sql"INSERT INTO ARTISTS(STAGENAME, AGE) VALUES(${obj.stageName}, ${obj.age})".update.run
    val tx: IOLite[Int] = getTransactor(q)
    tx.unsafePerformIO
  }

  override def update(obj: Artist): Int = {
    val q = sql"UPDATE ARTISTS SET STAGENAME = ${obj.stageName}, AGE = ${obj.age} WHERE ARTISTID = ${obj.artistID}".update.run
    val tx: IOLite[Int] = getTransactor(q)
    tx.unsafePerformIO
  }

}

This class has the necessary implementation of every operation and the logic for accessing the database and retrieving the necessary data. Please ignore (for the moment) all the concrete implementation with doobie because it'll be another post :)

And Reader?

We've seen so far three concepts: Algebras, interpreters and repositories. And I've said that Reader is a perfect fit for dependency injection.

Let's return to the scenario of dealing with multi-db support: how can we inject the correct repository without the magic tricks of DI like Guice or Spring Framework (IoC)?

Maybe you're wondering why are we bypassing all these solutions that makes our lives more easy? Well, we'll see the functional advantages that this monad give us (in fact, I gave you one reason before: composition. Refer to the [algebra section again]# The Algebra).

Let's review our ArtistService again:

trait ArtistService[T] {

  def listArtists(): Reader[ArtistRepository, List[T]]
  def retrieveArtistById(id: Int): Reader[ArtistRepository, Option[T]]
  def retrieveArtistByName(name: String): Reader[ArtistRepository, Option[T]]
  def insertArtist(artist: T): Reader[ArtistRepository, Int]
  def updateArtist(artist: T): Reader[ArtistRepository, Int]
}

Note that the return type of every function exposed by our service is a Reader. This monad takes two parameters: the dependency that we want to inject and the type that the function expects to evaluate to.

You can see Reader as a function that goes from A to B. Something like f: A => B. The difference here is that the function is somewhat lazy. I mean: the function is encapsulated in a monadic container (Reader) and you need to:

1. Provide the neccesary dependency.
2. Explicitely run the function to evaluate it.

Let's take a look at our service interpreter (I'll take only one operation for the sake of brevity):

class ArtistServiceInterpreter extends ArtistService[Artist]{

  override def listArtists(): Reader[ArtistRepository, List[Artist]] = Reader {
    repo: ArtistRepository =>
      repo.query()
  }
...
}

object ArtistServiceInterpreter extends ArtistServiceInterpreter

Note that this function takes the dependency (our repository) and executes the database operation needed. At this point, we haven't evalued the result, for that reason we need to provide the dependency and run our reader.

Our microservice is based on Akka-Http, so let's see how we're invoking a service operation based on our API definition:

import io.github.alejandrome.algebra.interpreter._
def api(repository: ArtistRepository): Route =
    handleExceptions(exceptionHandlerApi){
      handleRejections(genericRejectionHandler){
        pathPrefix("api") {
          pathPrefix("artists") {
              parameter("id".as[Int]){ id =>
                (pathEndOrSingleSlash & get){
                  createHttpResponse(
                    statusCode = StatusCodes.OK,
                    entity = ArtistServiceInterpreter.retrieveArtistById(id).run(repository).asJson
                  )
                }
              }
              ... 
          }
        }
      }
  }

Let's see this with more detail:

1. Repository parameter: our API need an ArtistRepository to run. We'll see shortly how we're providing the implementation.
2. Interpreters import: the top import statement is bringing to the scope all the service interpreters available. In our case, we're taking advantage of our created instance of ArtistServiceInterpreter to call the required operation to complete this request.
3. .run(repository): as we've said before, you need to explicitely run the computation contained into the Reader to get the value. That statement takes the dependency needed for materialize the computation. Short and sweet.

Look how easy is to call the required concrete service (interpreter) and injecting the repository for running:

ArtistServiceInterpreter.retrieveArtistById(id).run(repository)

Isn't that great?

Finally, let's take a look to our Boot class:

object Boot extends App with Api{

  implicit val system = ActorSystem("readerm")
  implicit val materializer = ActorMaterializer()
  implicit val executionContext = system.dispatcher
  implicit val logger = LoggerFactory.getLogger(this.getClass)

  val apiHost = system.settings.config.getString("akka.http.host")
  val apiPort = system.settings.config.getInt("akka.http.port")

  val akkaHttpLogger = Logging(system.eventStream, "readerm")

  val repo: ArtistRepository = ApplicationConfig.activeDatabase match {
    case "postgres" => new ArtistPGRepository
    case "h2" => new ArtistH2Repository
  }

  Http().bindAndHandle(handler = api(repo), interface = apiHost, port = apiPort) map {
    binding =>
      akkaHttpLogger.info(s"Readerm API Bound to address ${binding.localAddress}")
  } recover{
    case ex =>
      akkaHttpLogger.error(ex, "Failed to bind Readerm API to {}:{}", apiHost, apiPort)
  }

}

And our configuration file:

postgres{
  driverClassName = "org.postgresql.Driver"
  connectionString = "jdbc:postgresql:postgres"
  userName = "postgres"
  password = ""
}

h2 {
  # Pending implementation
}

activeDatabase = "postgres"

In this case, we're creating the neccesary database repository implementation upon starting our application based on a configuration that states what database engine we're going to use. The materialization is here:

val repo: ArtistRepository = ApplicationConfig.activeDatabase match {
    case "postgres" => new ArtistPGRepository
    case "h2" => new ArtistH2Repository
}

With that we're ensuring the creation of the correct dependency and its further injection.

Conclusions

It's a long post, I know, but I wanted to illustrate as detailed as possible these patterns and its implementation. If you have time, please take a look at the three further sections of this post in which I detail some considerations about this implementation.

So far, we've covered:

• The algebra and interpreters and its benefits from a functional point of view.
• The repository pattern, its advantages and an example implementation.
• Reader monad for dependency injection and how we can express computations based on its monadic properties.
• The combination of all this topics in a working implementation.

Remember to take a look at the project.

This approach succeeded in our real-world use case and enabled us to understand how these functional capabilities can make our software more expressive and maintainable.

Things to note

When building this project, I faced a couple of issues that I want to document here, in case that you encounter it:

• When you run a Reader, it evaluates to Id[T] instead of T. That was strange for me because I've used Kleisli before and with that abstraction the same operation (run) gave me a T. I asked in Stackoverflow and here is the answer for that behaviour.
• When I was learning how to use doobie I faced a behaviour in where IntelliJ wasn't capable of recognize the .query combinator. I lost a lot of time double-checking if I done something wrong and I came across with this and this. So, the conclusion is that there's a bug with IntelliJ and the sql interpolator. If you see that there are errors in ArtistPGRepository when using IntelliJ, don't worry, it's an unsolved bug of the IDE, and not of the project.

Naivety of this first implementation

I'm thinking about iterate this project with new improvements and to use it as a sandbox in where I can test new libraries / patterns. So I'm deciding to use it as the codebase for future posts here.

There are some issues at this time that I have to solve:

• Error handling.
• Implementation of the rest of entities of my domain.
• General refactors.
• Add more unit tests.
• Improve the doobie's Transactor implementation.
• Automate schema creation and data population.

So, you'll see some things that could make you think about the naivety of the implementation, but it is the first version.

What do we do next?

I had a great time playing around with doobie and I think that it deserves a separate blogpost.

Also I want to improve the model adding more complex operations to the algebra for practically demonstrate the compositionality that one can get of this approach.

For now, I will have my next post about doobie as soon as possible.

Thanks for reading! :)