krawler-meteofrance
A Krawler job designed to scrape data from the Météo-France public API
Overview
A Krawler based service to download diverse datasets from the public Météo-France API portal.
paquetobs
The paquetobs job files enable the scraping of real-time observations from all weather stations of the French network at hourly and/or sub-hourly intervals (every 6 minutes). These data are exposed by the public API Package Observations.
paquetobs-stations
This job scrapes the stations. The main collected properties are:
- id, the id of the station
- name, the name of the station
- location, the location of the station
The station data are stored in compliance with the GeoJSON standard.
Configuration
| Variable | Description |
|---|---|
PAQUETOBS_TOKEN | The token to use the paquetobs API |
DEPARTMENTS | The list of departments used to filter the collected observations, e.g, "11,09,31" |
STATIONS | The list of station ids used to filter the collected observations, e.g, "09099001,09301001" |
DB_URL | the database url. By default: mongodb://localhost:27017/meteofrance. |
DEBUG | Enables debug output. Set it to krawler* to enable full output. By default it is undefined. |
paquetobs-observations
This job scrapes the observations. The basic collected properties are:
- id, the id of the station
- name, the name of the station
- temperature (°),
- humidity (%),
- wind direction (°),
- wind speed (m/s),
- precipitation (mm/h)
The observation data are stored in compliance with the GeoJSON standard.
Configuration
| Variable | Description |
|---|---|
PAQUETOBS_TOKEN | The paquetobs API token. |
FREQUENCY | The frequency of the observations to collect. It must be horaire or infrahoraire-6m. Default value is horaire. |
LATENCY | The latency of the observations to collect. It must be horaire or infrahoraire-6m. Default value is horaire. |
TTL | The retention period in seconds of the data. By default: 7 * 24 * 60 * 60 (~7 days) |
DB_URL | the database url. By default: mongodb://localhost:27017/meteofrance. |
DEBUG | Enables debug output. Set it to krawler* to enable full output. By default it is undefined. |
Forecast Pipeline (AROME / ARPEGE)
These jobs download forecast data from the public Météo-France API:
Files are retrieved in GRIB2 format.
Available Jobs
| Job | Type | Resolution | Coverage |
|---|---|---|---|
arome-france | AROME | 0.025° | France |
arome-france-high | AROME | 0.01° | France |
arpege-europe | ARPEGE | 0.1° | Europe |
arpege-world | ARPEGE | 0.25° | Global |
Configuration
| Variable | Description |
|---|---|
DATA_SOURCE | Source of the data (meteofrance or data-gouv) |
AROME_TOKEN | API token for AROME |
ARPEGE_TOKEN | API token for ARPEGE |
OUTPUT_DIR | Base directory for downloaded files |
WORKERS_LIMIT | Maximum number of concurrent downloads |
OLDEST_RUN_INTERVAL_MS | Max allowed run age |
RUN_TIMES | Override model run times |
PACKAGES | Override packages |
FORECAST_TIMES | Override forecast steps |
Deployment
We personally use Kargo to deploy the service.