Using dask
Climetlab dask tools
You do not need CliMetLab to use dask. Dask is an independent python package often use with. to xarray and pandas. As CliMetLab seeks a strong integration to dask, some high-level functions may be useful to ease the interaction with dask.
Todo
This whole part about dask is EXPERIMENTAL, it will change in the future, may be removed or (very likely) moved to another package.
Start an local cluster and client
from climetlab.utils.dask import start
start('local')
# or $ climetlab dask start local
# or $ climetlab dask local --start
Start an ssh cluster and client
from climetlab.utils.dask import start
start('ssh')
Start a SLURM dask cluster and client
from climetlab.utils.dask import start
start('slurm')
Start a slurm dask cluster on HPC
This is assumes that your HPC admin set up the hpc-name-config-1.yaml file on the appropriate location.
from climetlab.utils.dask import start
start('hpc-name-config-1')
Access the dask dashboard
Todo
Not implemented yet.
Access the dask logs
Todo
Not implemented yet.
Stop the dask cluster
The dask cluster and client will usually stop automatically when the python process ends. Nevertheless, it is possible to stop dask if it has been started from climetlab.
from climetlab.utils.dask import stop
stop()
Advanced dask usages
Note: In this section a “dask deployement” refers to a client and a cluster. It does not refers to a Cloud deployement using Kubernetes, etc.
Create a custom dask deployement specifications
Create the yaml file $HOME/.climetlab/dask/hpc-name-config-1.yaml. Then use it with:
from climetlab.utils.dask import start
start('hpc-name-config-1')
Todo
This is EXPERIMENTAL.
Note
For HPC system admin: Adding yaml files in /opt/climetlab/dask/*.yaml will give global access to all users.
Reuse the dask client
from climetlab.utils.dask import start
client = start('local').client
Scale the dask cluster
Todo
Define what “scale” mean in this context.
from climetlab.utils.dask import start
deploy = start('slurm')
deploy.scale(..)