Kargo

# Getting Started

Using kargo is quite easy and basically the approach to setup your Geospatial platform is the following:

In the following guides, we will explain how to deploy a complete set of services on a given infrastructure such as illustrated in this diagram:

kargo-example-assumptions

Thus, we assume you already have a running Swarm infrastructure with:

  • A manager node
  • Multiple worker nodes with sufficient disk space to store the data

Check your infrastructure using the command docker node ls, you should have a response similar to:

$docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
jqtuxajexk18ypiylk6i6dv0q     worker-0            Ready               Active                                  18.03.1-ce
x6tc2sqi99vp106icwf2nmyp0 *   manager             Ready               Active              Leader              18.03.1-ce
wfr5hwhfd5413p2ql9hwitbkw     worker-1            Ready               Active                                  18.03.1-ce
8980x3d76x1r7kxoa7h5lzob8     worker-2            Ready               Active                                  18.03.1-ce

Prerequisites

If you do not have such an infrastructure, you may have a look at Kaabah (opens new window).

# Prepare the infrastructure

# Install the prerequisites

# Install rclone

rclone is used to provision the nodes with the static data that are used by the services (i.e. geospatial datasets, docker images...). rclone allows you to retrieve these data from most of the well know stores such as AWS S3, OpenStack Object Store, Google Drive. You need to install rclone on each worker:

  1. Install rclone
sudo apt-get install rclone
  1. Configure rclone

Use the following guide (opens new window) to setup rclone

# Install jq

jq (opens new window) is a lightweight and flexible command-line JSON processor. The CLI uses jq to extract some information from the package.json file.

On the manager, type the following command:

sudo apt-get install jq

# Install yq

yq (opens new window) is a lightweight and portable command-line YAML processor. The CLI uses yq to merge configuration files.

Please refer to the installation section (opens new window) to install yq

# Install GlusterFS

To use Kargo, it is required to share the configuration among the nodes of the swarm. To create a shared filesystem, we recommend to use GlusterFS (opens new window). Please refer to the documentation (opens new window).

# Install kargo

  1. Clone the Kargo repository

On the manager node and clone the repository in the shared directory of your choice:

$git clone https://github.com/kalisio/kargo.git

# Provision the data

It is up to you to copy your data to the different nodes. You must have to keep in mind that these data will have to be accessible by the services that you want to deploy.

Looking for data ?

We can provide datasets from different sources such as public catalogs and those of our partners such as OpenMapTiles (opens new window), PlanetObserver (opens new window). Do bot hesitate to contact us at contact@kalisio.com.

# Setup the services

# Setup your workspace

  1. Create a directory to store your configuration stuff. Give it the name of your choice.

  2. Create the Environment file by making a copy of the .env.sample file into your configuration. Rename it as .env

  3. Edit the .env file and configure the services you want to deploy. See the section Environment to have the complete reference of the services settings.

  4. Define node constraints. For each node, define a variable <NODE_NAME>_LABELS listing the labels to be assigned to the node

MANAGER_LABELS=""
WORKER_0_LABELS="tileservergl=true mongodb=true"
WORKER_1_LABELS="tileservergl=true"

TIP

To check you node labels mapping, use the following command:

$docker node ls -q | xargs docker node inspect -f '{{ .ID }} [{{ .Description.Hostname }}]: {{ .Spec.Labels }}'

TIP

The labels you define on nodes can be used as placement constraints for the services you'll deploy, see here

  1. Define the stacks you want to deploy:
    1. Define the STACKS variable listing the stacks
    2. For each stack, define a variable <STACK_NAME>_STACK listing the services to deploy
STACKS="dbs weacast hydro air apps"
APPS_STACK="tileservergl mapserver mapcache maputnik thredds kano"
DBS_STACK="mongodb"
WEACAST_STACK="arpege-world-loader arpege-europe-loader arome-france-loader gfs-world-loader weacast"
VIGICRUES_HYDRO="vigicrues hubeau-stations hubeau-observations"
AIR_STACK="teleray openaq"
  1. Tell kargo to use your workspace
$./kargo use <path/to/your/workspace/directory>

TIP

We strongly recommend to use Git to manage your workspace. If using Git, you can take advantage of the command kargo pull to pull the workspace.

  1. Configure Kargo
$./kargo configure
  1. Create the constraint labels on the nodes
$./kargo labels add

# Build the services

Some of the services provided By Kargo need to be built and pushed to the local registry in order to be able to deploy them. You simply need to run the command:

$./kargo build <service>

# Manage the services

# Getting information

You can retrieve some information about the current configuration using the command info

$./kargo info

# Deploy the services

You can either deploy a given stack or deploy all the stacks.

  • Deploy a stack
$./kargo deploy <stack>
  • Deploy all the stacks
$./deploy deploy all

# Remove the services

You can either remove a given stack or remove all the stacks.

  • Remove a stack
$./kargo remove <stack>
  • Remove all the stacks
$./remove remove all

← Advanced usage →