As you probably know, I’m a big fan of Akka. Akka Cluster is a great low-level mechanism for building reliable distributed systems. It’s shipped with powerful abstractions like Sharding and Singleton.
If you want to start using it you should solve cluster bootstrapping first. Almost every tutorial on the internet (including the official one) tells you to use seed nodes. It looks something like this:
1 2 3 4
but wait… Hardcoding nodes manually? Now when we have Continuous Delivery, Immutable Infrastructure, tools like CloudFormation and Terraform, and of course Containers?!
Well, Akka Cluster also provides programmatic API for bootstrapping:
So, instead of defining seed nodes manually we’re going to use service discovery with Consul to register all nodes after startup and use provided API to create a cluster programmatically. Let’s do it!
You can find the demo project here: https://github.com/sap1ens/akka-cluster-consul. It’s very easy to try (with Docker):
Docker Compose will start 6 containers:
- 3 for Consul Cluster
- 3 for Akka Cluster
Everything should just work and in about 15 seconds after startup you should see a few
Cluster is ready! messages in logs – it worked!
More details below ;–)
First of all I want to explain what this demo service does.
I implemented simple key-value in-memory storage with the following interface:
1 2 3 4
So we can set, get and delete string values. All these actions are exposed via HTTP API:
- GET http://localhost/api/kv/$KEY
- PUT http://localhost/api/kv/$KEY/$VALUE
- DELETE http://localhost/api/kv/$KEY
This simple functionality is going to be started as a Cluster Singleton. It means that Akka Cluster will make sure to keep only one copy of the service running. If the node with the service fails Akka Cluster will start the service in another node and forward all in-flight messages there. So it’s great for high availability and resiliency.
Service Discovery with Consul
To be able to use Consul we need to implement two things:
- Service registration. Every node with a running service should report about itself to Consul. Usually it’s done using Consul Agents, but in our case we’re going to use Consul HTTP API for simplicity. We use a shell script to simply hit Consul
/v1/catalog/registerAPI endpoint with required metadata and run the app.
- Service catalog. We need to fetch a list of service nodes before calling
joinSeedNodesmethod. Our app uses OrbitzWorldwide/consul-client Java Client, so resulted code is very straightforward:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Our Consul UI will look like this in the end:
Akka Cluster Setup
Akka Cluster needs quite a few configuration settings, but the most important are these two:
min-nr-of-members guarantees that cluster won’t be created until at least N nodes are joined.
1 2 3 4 5 6 7 8 9 10 11 12
akka.remote.netty.tcp contains networking configuration for our cluster, which looks a bit complicated (but it should make more sense when you look at the second part of the Docker Compose file).
So, Akka Cluster needs dedicated TCP port for its gossip protocol. Since we run multiple Docker containers on the same machine we have to dedicate different ports for different containers. In our case first node will use port 80 for HTTP API and port 2551 for gossip, second node will use port 81 for HTTP API and port 2552 for gossip and the third node will use port 82 for HTTP API and port 2553 for gossip, accordingly. In production you might simplify it if every service is running on a separate machine, you just need one port everywhere.
More explanation about port and hostname values can be found here.
Finally, let’s look at the cluster bootstrapping logic:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
Here we create a scheduler to retry our attempts every 30 seconds with initial 10 seconds delay. It can be useful if Consul is not immediately available. Then we receive a list of service addresses from Consul and convert them to a list of seed nodes, using a few simple rules:
- Consul always returns nodes in the same order
- First node will always join with itself to form the initial cluster. Read a comment for more details
When cluster is established,
init method is called. It contains all cluster-specific logic, like creating Singletons or Sharding. In our case it creates Cluster Singleton for our KVStorageService:
1 2 3 4 5 6 7 8 9 10
So when you see
Cluster is ready! message you can try the HTTP API! You can use any of our nodes (localhost:80, localhost:81, localhost:82), any node will know where Singleton instance is located and forward request-response if needed. Magic :)
As you can see, this setup is not particularly challenging, but it gives great benefits – we shouldn’t worry about bootstrapping cluster manually anymore. Consul is great for starting with service discovery, so this approach should be safe for introducing service discovery to your system.
Our solution will also work for rolling deployments and cluster resize.