Understanding Kaabah

Prerequisites

Kaabah relies on various technologies such as Terraform, Docker Swarm, Traefik... and we assume that you are enough familiar with them. If not, please take a while to discover them.

Key concepts

Kaabah let you manipulate 4 kind of entities:

  • Workspace: a collection of everything Kaabah needs to create and manage an infrastructure.
  • Configuration: a set of Terraform variables used to design your infrastructure.
  • Cluster: a Docker Swarm built using Kaabah.
  • Service: an application deployed on your Cluster. By default, Kaabah comes with the following services which helps operating the Cluster:

The following image illustrates how these entities interact:

Kaabah principle

In addition provides a set of commands that help you to operate the cluster. For instance, you can easily prune all the images on the different nodes, execute a command on a given node... See the reference page to have the complete list.

Note

The helper commands are installed in /usr/local/sbin and can be used out of the box.

Workspace

Kaabah is designed to take advantage of Terraform Workspaces. Indeed, Kaabah relies on the Terraform recommend practices and assume a Workspace is used to store the required data needed to build and manage an infrastructure for a specific environment (staging, production...):

  • the configuration of the infrastructure.
  • the SSH private key to get connected to the infrastructure
  • the extensions to the services you want to be installed by Kaabah
  • the user scripts you want to be executed when creating the infrastructure
  • the SSH private key to get connected to the infrastructure
  • the Terraform states of the infrastructure.

Starting from this premise, Kaabah lets you to manage as many clusters as your projects require. If we decide to name our workspaces with both the project name and its environment (i.e. dev, test...), we can sketch the following diagram to illustrate the overall functioning of Kaabah:

Kaabah terraform

In this diagram, the states of the different workspaces are stored within a dedicated bucket on amazon S3, but you are free to use any other Terraform backends.

Configuration

The Kaabah configuration file is a Terraform variable file describing the characteristics of the desired infrastructure.

Here is an example of a configuration file:

provider = "AWS"

manager_ip = "3.115.176.41"

manager_instance_type = "t2.small"

worker_instance_type = "t3.large"

worker_instance_count = 3

worker_additional_volume_count = 2

worker_additional_volume_size = 500

worker_additional_volume_type = "st1"

worker_additional_volume_mount_point = "DATA"

Assuming the current workspace is app-dev, then when applying such a configuration, Kaabah will generate a Docker Swarm infrastructure on AWS (provider variable) composed of:

  • one manager node, app-dev-manager, of type t2.small with the public IP address 3.115.176.41.
  • 3 worker nodes, app-dev-worker-0, app-dev-worker-1 and app-dev-worker-2, of type of t3.large. To each worker is attached 2 optimized hard-disk (sc1) of 500GB and these volumes are accessible through the mount points: /mnt/DATA0 and /mnt/DATA1.

Kaabah exposes many more variables allowing you to customize in detail your infrastructure such as specifying a domain name, adding labels, running user scripts... Have a look at the complete list of variables and the tests as an example.

Cluster

The cluster consists in of multiple Docker hosts which run in swarm mode and act as manager (to manage membership and delegation) and workers (which run services).

swarm concept

Instances

When generated from a given <WORKSPACE>, the instances are named according the following convention:

  • <WORKSPACE>-manager
  • <WORKSPACE>-woker-<INDEX>

You can add some labels to the nodes using the manager_labels and worker_labels variables.

Volumes

When needed extra disk spaces, you can attach additional volumes to the workers. These volumes are automatically attached, formatted (ext4 file system) and mounted on the workers. By default the volumes attached on a worker are accessible with the paths /mnt/data0, mnt/data1 and so on. You can override the default data mount point by overriding the worker_additional_volume_mount_point variable.

Security Groups

By default, Kaabah creates 2 security groups:

  • the manager security group allowing:
    • SSH traffic (port 22).
    • HTTP traffic (port 80).
    • HTTPS traffic (port 443).
    • Docker swarm traffic. The traffic is limited to your private network.
  • the workers security group with the following rules:
    • SSH traffic (port 22)
    • Docker swarm traffic. The traffic is limited to your private network.

Docker network

The traffic between the nodes is relying on a Docker network of type of Overlay.

The name of the Docker network is automatically computed from the workspace name but you can override it using the docker_network variable.

Security

In Swarm mode, the nodes communicate using an HTTP socket and therefore it is highly recommended to protect the the Docker daemon. Kaabah will do it for you ! Therefore the Docker daemon only allows connections from clients authenticated by a certificate signed by a Certificate Authority (CA).

When creating the cluster, Kaabah handles the creation of the server and client keys but it requires you to provide this CA. Check out the Getting started section to learn how to generate this CA.

Note

Kaabah relies on OpenSSL to generate the server and client keys.

Services

As mentioned in the introduction, Kaabah bootstraps a cluster with a stack of high level services that allows you to:

  • route the traffic to the cluster and ensure SSL termination using Traefik
  • manage the services deployed on the cluster using Portainer
  • monitor the cluster using Prometheus
  • analyze the cluster metrics using Grafana

Note

The access to the UI of the different services is protected using Basic Authentication.

In addition, Kaabah lets you extend this stack to add the services of your choice. See the Extending the services section to learn how to do it.

Traefik

traefik allows to route the traffic from internet to the Docker Swarm infrastructure with SSL termination. It uses Let's Encrypt to generate and renew SSL certificates for each services.

WARNING

Du to rate limits fixed by Let's Encrypt, it is highly recommend to set the ca_server variable to https://acme-staging-v02.api.letsencrypt.org/directory when testing your infrastructure.

By default, Kaabah specializes the traefik configuration with:

  • 2 entrypoints:
    • to allow HTTPS requests (port 443)
    • to redirect HTTP (port 80) request to HTTPS (port 443)
  • 5 frontends to access the services:
    • traefik (dashboard)
    • Portainer
    • Prometheus
    • Alertmanager
    • Grafana

The frontend rules depend on the subdomain and donain variables defined in the Terraform configuration. Considering a workspace named app-dev, the default subdomain will be app.devand the traefik configuration will be as shown in the following diagram:

traefik routing

Portainer

There is not too much to say on Portainer. Just play with the UI !

Prometheus

The Prometheus monitoring solution is shipped with the following components:

Grafana

Grafana allows you to visualize the metrics stored in the Prometheus databases through dedicated dashboards.

By default, Grafana includes the following customizations:

  • UI:
    • Login form disabled: indeed the access to Grafana requires to be authenticated (basic auth). This requirement is defined using Traefik frontend rule.
    • Once authenticated, the default user is granted the EDITOR permissions.
  • Datasources: the Prometheus datasource is included by default.
  • Dashboards: 2 dashboards are provided by default:
    • Cluster overview which allow to visualize the main metrics of the cluster nodes
    • Swarm overview which allow to visualize the main metrics of the Docker swarm

Registry

The Registry helps you build local images on your cluster and deploy them as a service.

Read the Using the Registry section.