A Story of a Microservice: Lessons from the Trenches

A lot has been written about microservices over the years, but we feel that not many of these articles have presented real-life and long-term experiences of building and maintaining microservices. In this blog post we aim to address this shortcoming.

Microservices are loosely coupled, independently deployable applications that are focused on fulfilling a single cohesive responsibility. The microservices mindset encourages continuous deployment cycles, promotes choosing the right tool for each job, and helps to build a highly fault-tolerant architecture that can be evolved and scaled on a fine-grained level. Implementing a microservice architecture requires a substantial investment in an automated deployment infrastructure.

For the past three years, Reaktor has played an instrumental part in building a microservice architecture for Yleisradio (Yle), Finland’s national broadcasting company.

This architecture serves as the foundation for Yle’s online crown jewels: popular news websites, the Yle Areena streaming service (think national Netflix), and the award-winning Uutisvahti mobile news app, to mention just a few examples.

The polyglot architecture consists of a few dozen microservices, many of which implement RESTful APIs that power the aforementioned services. In production, the architecture utilizes hundreds of computing instances that handle over three billion requests each month. Production deployments are frequent – on a typical day, tens of deployments are made by twice as many developers.

Microservices == freedom

In late 2014, Yle’s existing API Authentication and Authorization Service (AAS) provider tacked another zero onto the end of their monthly bill because our usage had increased dramatically as Yle’s APIs had come into more widespread use.

This meant we had to find a replacement – fast.

The new service would have to be able to support existing credentials, provide throttling and usage statistics, be highly available and easily maintainable, and to handle millions of daily requests. Next, we began exploring both open source and commercial closed-source solutions that could fit the bill.

We were facing a Goldilocks dilemma: All the existing products were either too large, containing everything but the kitchen sink – which would bring on unnecessary complexity – or too small, because they didn’t have the features we required. The solution that proved just right for us was to build an AAS microservice ourselves.

So one day, on the train to the clojuTRE conference, the Reaktor team hacked together a proof-of-concept based on nginx-clojure, a library that combines two proven technologies: the nginx Web server and the Clojure programming language.

Build the simplest thing that solves your problem

The clock was ticking – the existing service’s support contract was soon up for renewal.

Having hacked together the proof-of-concept, we proceeded to build the new service in small, immediately deployable increments. Bringing the proof-of-concept to production-level quality took two developers roughly four weeks.

Initially, the user credentials were hard-coded in the application’s source code. In the spirit of building the simplest thing that would possibly work, we extracted the credentials to a simple flat file which we brought under version control. This allowed us to avoid the complexity brought on by having to maintain a fully-fledged database installation.

During the next increments, we implemented throttling and other features. After each iteration, we ran a suite of performance tests to ensure that the service was performing to our requirements.

The new and the old authentication systems were ran in tandem until we felt comfortable that the new service was stable and feature-complete enough to replace the old one. Another welcome upside of the new service was that the average response time dropped from 200 milliseconds to around 0.1 milliseconds.

Simple solutions save money over time

Fast forward two years: The number of microservices and users guarded by the AAS microservice have multiplied many-fold, as has the amount of traffic that goes through it.

Some new features have been introduced, but the codebase has remained small and easy to reason about. The implementation has stood its ground even now when the infrastructure has started to migrate from Yle’s private cloud to Amazon Web Services (AWS).

The proof is in the pudding: A simple solution saves you money over time by keeping the maintenance costs low.

All in all, don’t be afraid to start off small and experiment. Since you have an automated deployment infrastructure in place, you can get your prototypes up and running quickly and with little additional effort.

On the other hand, if it turns out that you’ve built the wrong thing, it’s not like you spent countless years and oodles of money on developing it – you can throw it away and start over without much heartache.