How to build a map print service in minutes

This article introduces the minimalist ETL krawler through the practical use case of creating a web service to print maps.

Preambule

At Kalisio we are used to build geospatial applications and a recurrent need of our customers is to be able to print some geographical entities on top of a map background. Previously, we deployed sophisticated solutions like MapFish Print or Mapnik for such a use case. Although doing well, the learning curve is really steep, customization not so easy and maintenance in the long-term difficult (required skills, deployment issues, etc.). You often have to cope with the fact that you develop the same things twice: first in a web client for instant visualization and second in a print service.

Recently we have started an effort to develop a minimalist Extract, Transform, Load (ETL) tool called krawler. It allows to run tasks (a.k.a. a job) flowing through a pipeline: a set of processing functions connected in series. Krawler comes in different flavors: as a CLI or as an API. A detailed description of the internals of krawler are available in the following Medium articles:

• A minimalist ETL using FeathersJS — Part 1
• A minimalist ETL using FeathersJS — Part 2

Last but not least, we have started using TestCafé to perform end-to-end testing of our own web applications with ease.

Gluing everything together we imagined a new solution for a simple map print service. Indeed, our customers are often satisfied with a print service which output is similar to what they would get by taking a screenshot of their web mapping client once correctly setup. So here is what we planed to implement:

1. a simple web mapping client with required features
2. a user simulation performing layer selection, data import and screenshot
3. a web service providing an API to automate everything

Building the web mapping client

This is actually the easiest part, we built a plain vanilla JS Leaflet-based web client, including all required features with a couple of plugins (base map selection, overlay selection, GeoJson data import, print), you can see it in action here.

A simple web mapping client with data layer selection, data import and print features

To configure the print size you can either print the current view, so that the output is driven by your screen resolution, or by selecting a predefined format like A4 Landscape defined in pixels at 90 DPI by default in the print plugin (but you can add your own format if required).

The underlying map data infrastructure is powered by Kargo

Simulating the user printing his map

The easiest way to start with TestCafé is by using the official Docker image. Indeed it plays well with headless environments by either running headless browsers or behind Xvfb. We simply customized it by integrating our test code to simulate the user, here is our Docker file (quite simple isn’t it ?):

FROM  testcafe/testcafe
WORKDIR /opt/testcafe
COPY ./src /opt/testcafe/src

Build the image like this:

docker build -t kalisio/leaflet-print .

Then we created the page model of our client using selectors and methods to perform all required actions:

In a production code you have to tweak timers to ensure a good performance/quality ratio since map data are always loaded asynchronously. More recently a new HTTP requests interception feature has been added to TestCafé that could also be used to detect if data download is finished. The print plugin has also some options you can play with.

The user simulation code simply consists in calling all the page model methods to build the print scenario:

Because everything will run in a Docker container we manage the dynamic configuration of the simulation (e.g. the base layer to be used) through environment variables. To import custom GeoJson data we also need to put the file inside the container by copy or volume mounting, then to get back the output printed image by copy as well. Using the Docker CLI:

// Perform print
docker run --name printer -e "GEOJSON_FILE=myfile.json" -v /home/data:/tmp kalisio/leaflet-print 'chromium --no-sandbox' /opt/testcafe/src/index.js
// Retrieve the print output and terminate the container
docker cp printer:/home/user/Downloads/map.png .
docker rm -f printer

An output of the print feature after running the user simulation on the web mapping client with a simple GeoJson layer

Creating a web service to automate everything

In order to get things work automatically by calling a web service API we need to setup a pipeline to perform all the required tasks, helpfully krawler is exactly what we need. In krawler there are two main abstractions: hooks (i.e. processing functions) and services to manipulate three kind of entities:

1. stores define where the extracted/processed data will reside,
2. tasks define what data to be extracted/processed/loaded and how,
3. jobs define what tasks to be run to fulfill a request.

In our use case we will use a local file system (FS) store for intermediate files copied to/from the container and a AWS S3 store to output the final printed image. A task will be a print scenario execution and a job a set of print tasks to be run. A task will include the GeoJson data to be used as well as the print parameters, to access them in processing functions we will make intensive use of options templating allowed by krawler. The following mapping between task properties and environment variables will be used to send parameters to the container:

• appUrl <> APP_URL: URL of the target web client
• baseLayer <> BASE_LAYER: name of the base layer to use
• overlayLayer <> OVERLAY_LAYER: name of the overlay layer to use
• width <> SCREEN_WIDTH: screen width for custom print size
• height <> SCREEN_HEIGHT: screen height for custom print size
• format <> PRINT_FORMAT: predefined print format to use
• data <> GEOJSON_FILE: name of the file automatically produced from input raw data

To avoid submitting the required internals for a job (e.g. the output store), or the default values for a task (e.g. the target app URL), we also apply a template to automatically add all information on each request so that the client has nothing special to do to deal with. Similarly we apply a template when the job is done to output the desired fields in the response for each task, in our case a download link on AWS S3 for the output image. The simplified version of the pipelines is the following:

Hooks diagram for the print job and print task pipelines (generated using mermaid)

It omits some implementation details like the generation of a UUID for each task to avoid file name collision or the fact that the Docker API can only copy tar archives to/from containers and not files directly. The complete job file can be found here, to run it as a web service:

krawler jobfile-print.js --api

By default krawler launches the web services on port 3030 with an api prefix, so that you have to POST this request on localhost:3030/api/jobsto make it work:

{
 "tasks": [{
  "format": "A4 Landscape",
  "baseLayer": "PlanetSAT & OpenStreetMap",
  "data": {
   "type": "Feature",
   "geometry": {
    "type": "LineString",
    "coordinates": [[102.0,0.0],[103.0,1.0],[104.0,0.0],[105.0,1.0]]
   }
  }
 }]
}

The response will look like that:

[
    {
        "id": "bc185cb0-7983-11e8-883f-a333a7402f4a",
        ...,
        "link": "https://s3.eu-central-1.amazonaws.com/krawler/bc185cb0-7983-11e8-883f-a333a7402f4a.png"
    }
]

You can now download the image. If you’d like to print images with a set of different map backgrounds or different data simply POST a request like this:

{
 "tasks": [{
  "format": "A4 Landscape",
  "baseLayer": "PlanetSat",
  "data": {
   "type": "Feature",
   "geometry": {
    "type": "LineString",
    "coordinates": [[102.0,0.0],[103.0,1.0],[104.0,0.0],[105.0,1.0]]
   }
  }
 }, {
  "format": "A4 Landscape",
  "baseLayer": "PlanetSAT & OpenStreetMap",
  "data": {
   "type": "FeatureCollection",
   "features": [{
    "type": "Feature",
    "geometry": {
     "type": "Point",
     "coordinates": [102.0,0.5]
    }
   }, {
    "type": "Feature",
    "geometry": {
     "type": "Point",
     "coordinates": [103.0,0.5]
    }
   }]
  }
 }]
}

Conclusion

We don’t want to say here that our solution is more relevant than well-known and powerful tools dedicated to print maps. More specifically we did not address desirable things like PDF generation, high-resolution printing, etc. However we hope that for the sole goal of automating map images generation our toolkit has proven to be a more simple and extensible approach. The total amount of code for the user simulation and the web service configuration is less than 200 lines ! We also believe that aesthetic stuffs like map legends, scale controls, and so on, are more easy to manage in the web client. The full code of this article has been added to the latest krawler documentation.

If you liked this article feel free to have a look at our Open Source solutions, the Kalisio team !