Installing Krawler

As Command-Line Interface (CLI)

npm install -g @kalisio/krawler
// Or with Yarn
yar global add @kalisio/krawler

As module

As dependency in another module/app:

npm install @kalisio/krawler --save
// Or with Yarn
yar add @kalisio/krawler

Or when developing:

git clone https://github.com/kalisio/krawler
cd krawler
yarn install
yarn link
// Now you can launch the krawler CLI
krawler jobfile.js
// Or link wirh it in another package
yarn link @kalisio/krawler

Please refer to the KDK documentation (opens new window) to setup your development environment.

A native command-line executable can be generated using pkg (opens new window) eg for windows:

pkg . --target node8-win-x86

Because it relies on the GDAL native bindings you will need to deploy the gdal.node file (usually found in node_modules\gdal\lib\binding) to the same directory as the executable. Take care to generate the executable with the same architecture than your Node.js version.

As a Docker container

When using krawler as a Docker container the arguments to the CLI have to be provided through the ARGS environment variable, along with any other required variables and the data volume to make inputs accessible within the container and get output files back:

docker pull kalisio/krawler
docker run --name krawler --rm -v /mnt/data:/opt/krawler/data -e "ARGS=/opt/krawler/data/jobfile.js" -e S3_BUCKET=krawler kalisio/krawler